Data Quote Streaming

Posted in IB E1 on June 24, 2010 by kpervushin

http://www.esignal.com/livecharts/pricing.aspx

http://www.xignite.com/stock-quote/real-time-stock-quotes-data–20003003.html (Real Time)
http://www.xignite.com/stock-quote/delayed-stock-quote-data–20003007.html (Delayed)
http://www.dukascopy.com/english/ddf_main/rdata/ExportCSV/ (Historical Data)

http://www.nasdaq.com/

http://stockcharts.com/

http://www.realtimestockquote.com/realtimestockquotechart.asp

http://www.forexcentral.net/fab/financial-data-providers (This page contains links for some financial data providers)

http://corporate.morningstar.com/US/asp/subject.aspx?xmlfile=6695.xml

The System Metaphore

Posted in IB E1 on June 23, 2010 by kpervushin

we employ so called Agile Software development method,
which means no “big design up front”, instead we use (i) user stories, (ii) test driven development based on something called (iii) the “system metaphor”. The system metaphor allows you to talk about your design in a consistent and unambiguous way (the same vocabulary, terms etc).

I suggest The system metaphor as follows: It is an educational multiuser social network (Wikipedia like) specifically catering for trader’s needs in the area of options trading with capabilities of actively suggest (construct), verify, publish (to members or public), initiate and finish option trades from fully manual to semiautomatic way according to the trader’s individual parameters or profile.

The functionality is described in the form of user stories, e.g. short descriptions of features to be implemented. They describe a small piece of functionality that can be broken down into tasks whose durations can be quickly estimated.
Story 1:
John is willing to join Trader’s Round Table and receive regular invitations to the events of the Club.
He wants to get access to educational videos and other material helping him to understand fundamentals of options trading, get useful tips on recommended Brokers and Internet resources.
He wants to know who are other members of the Club and potentially introduce himself to them and see their scores and activity levels (how many trades they made, what kind etc).
He wants to discuss certain issues related to broader aspects of trading.

// The above functionalities are typical for a social networking platform

Story 2:
Max after attending Thomas’ promotional lectures would like to try his trades on the paper account, however he is not yet certain how much of the capital he is ready to invest and how active he can be. The system shall be able to suggest him some trades and simulate order placement, and trade closing on the historical data.

// This will be an active functionality with graphical interface, which would relay on the historical market data and simple execution algorithms.

Story 3:
Jane has opened account with some Broker, but has no idea how to place orders and monitor trades. She specifies in her profile this information, and receives specific instructions (video) relevant to her situation. She might also appreciate some suggestions for her capital allocation in portfolio and set limits for individual trades (e.g. if her capital is 15000 USD, she must not risk more than 300 USD per trade) according to Thomas’ money managment rules.

// This is adjustment of the system to individual traders, building/management of individual portfolio and trade constraints according to your proprietary teaching system.

Story 4:

Jack is not yet sure what kind of trade he can attempt.. He wants to see what other students are planning or trading (for educational purposed them voluntarily disclose their real or paper trades).

etc..

Out of these short stories we have to build functionality list. This list will be the specification target for various phases and deadlines.

We have to also find a benchmark existing application(s). I can suggest (i) wikipedia.com (2) optionistics.com

Benchmark websites providing educational social network

Posted in IB E1 on June 23, 2010 by kpervushin

First of all, none of them are about creating a social network of traders sharing information. They are plainly selling whatever little they can and run away. Largely they are anonymous and can close the shop any time. We are going to create an educational platform with values shown upfront with or without subscription. We are not anonymous, we have physical location and real names. we create community, we are not selling bullshit.
As you can see there are not many of any webservices similar to our project.

www.TradersRoundTable.com.sg

www.TRT.com.sg

Those links do not work.

http://www.ecnprovider.com/

This one is such a hoax! I now very similar company in Singapore, founded by former staff of http://jflennon.com . They used to teach forex in Singapore. The jflennon was sold and the staff left with nothing. They decided to sell EA for Metatrader 4. Only complete idiots can buy into this (and I am sure there are some around). This doesnot come any close to what we want to develop.

http://coveredcalls.com

This site is a simplified version of optionistics.com (I am a subscriber to the latter one and to IVolatility.com).
This is a simple screener for overpriced calls (this simply means high IV). There is a small educational component in form of webinars. No community, no care of individuals.

From terrance:

http://www.10percentpermonth.com/Home.html

this is complety anonymous site as ecnprovider.com Their offer is dubious and claims unverifiable. It is not a benchmark for us at all.

http://www.fx360.com/

this is one of many fx oriented sites. Comparable and competing with dailyfx.com, forexfactory.com, oanda.com etc.
It provides important information on calendar events, futures open interests, expert opinions, forex streams, some education.

eSignal with IB

Posted in IB E1 with tags on May 28, 2010 by kpervushin

youtube video to bridge IB and eSignal

the trades placed at IB are not seen at eSignal.
eSignal activity is seen at IB.
Commissions on 100 000 USD are 2.9/2.5 USD for Buy and Sell. Commissions are applied to open and to close(cover).
100 000 USD is 1 Lot, e.g. 1 pip is 10 USD.
minimal is 20 000 USD (at IDEALPRO) and commissions still the same, e.g. 2.5 USD.

eSignal http://www.esignal.com
IB http://www.interactivebrokers.com

NMR language

Posted in NMR generator, NMR simulator with tags , on January 9, 2009 by kpervushin

I propose to create a better language for formal description of NMR experiments than say Bruker or Varian coding language. We will encode NMR experiments using this optimized universal syntax and store them in a web database for subsequent translation (at setup time) into Bruker coding for execution on a given real spectrometer or to Spinach for simulation. Thus, we can store only an blueprint of an experiment removing its immedeate implementational dependances. This will help to optimize individual experiments using theoretcal models.

NMR experiment consists of a basic timing of events diagram known generically as pulse sequence as such and a calibration convention connecting abstract entities as a 90deg pulse to shaped radio frequency burst of given strength (in db units) and time duration specific to a given spectrometer as well as some auxiliraly files like composite pulse decoupling sequences inserted as wholistic locked units within the basic time diagram.

A certain high level (generalized) representaion of experiments ideally should be human oriented with dynamically accessible context information (like spin state at this time or pulse). In one of my projects (submitted to Bruker, but rejected) we proposed even to generate NMR experiments directly from chemical structure, e.g. just mark spins to be correlated and get working experiment. At this time I was a member of Informatics Institute of Uni Zurich, people their liked this kind of global ideas and were ready to try to implement.

We should list requirements for such a language first, of course.

Well, after some thinking, I propose the radical solution — I will write a compiler that would take pulse sequence description and generate the effective Liouvillian stack. It seems there’s no other way than the one musicians are taking — they have a basic time grid and all events are snapped to it and discretized on it. The grid will be adaptive and will have a form of a vector of timings plus a stack of matrices. Run that through expv and we’re done.

server side Spinach

Posted in NMR generator, NMR simulator on January 6, 2009 by kpervushin

Dear Ilya,

nice minutes! Impressed. What kind of software did you use to produce them? ms project?

As to XML description of pulse sequences. An experiment contains the timeing events digaram (akin you gif picture) and auxiliray info like calibration tables and cpd, shapes etc files. The problem is that we need first to calculate a given experiment encoded in say Bruker or Varian library in Spinach in order to make meaningful interpretaion of events in timing diagram such as hard pulses to 90 degree, 180 degree or other categories. Only after that graphical rendering makes any sence. More over, such a graphical picture is intuitively clear only for simpliest HSQC or COSY like experiments. However, if you look at 3D experiments, the jumble of pulses and delays quickly makes traditional graphical representation completely usless. My point is that the experiment not nessessery shall be represented as actual timing events diagram, but we can move to more general logical LEGO blocks like transfer between this and that spin state. This would reveal the design behind the experiment, and in fact, would enable optimal construction of new experiments from the designed goals, like to correlate this spin state with this using LEGO styled blockes like INEPT. Very soon we have to move to optimization of the existing pulse sequences, so that all controls (phases, delays etc) should be accessible programmatically. What you are showing as XML stab, can be in fact a representational derivative of a higher order structure, but not the basis of experiment programming.

I have created a database of commonly used experiments encapsulating all files and procedures needed to setup an experiment of choice on any given spectrometer. I used CARA as an environment (www.nmr.ch) and LUA language (with lots of widget classes from CARA) to create GUI. I meant it to be lightweight, so that the whole database is a single XML file shareble via web. Soon I bumped into a problem of many concurrent users modifying entires in the database creating parallel versions of this file. We decided to move to server-side solution to maintain order in multiuser env and harniss the power of community to quickly debug the entire pool of experiments. My postdoc in Basel took one of the free wikis (written in .NET on C#) and started to hack server-side solution while I am doing desktop application. From here stemms my suggestion to use C#, – I have there experience and manpower already.

Ideally, all of us wish to have a united site with social networking possibilities, where we can expose community editable NMR experiments of any complexity with all aux files in a form which can self setup on any end user spectrometer (inspecting the local configuration if needed). Of course, Spinach fits neatly into this scheme providing a powerful analytical tool capable of semantic analysis and optimality checks (against fundamental quatum bounds) of the submissions followed in perspective by actual optimization.

I strongly recommend not to write GUI from scratch and even not to use David’s hacks. They might seem useful, but in fact, they will exist only in matlab environment as prototypes. Still better is to adopt an existing open source server side solution (wiki like, or more global content management systems, there are aplenty of them) equipped with CSS, styling etc and build GUI on the top of it right from the beggining.

As to David’s simulator. David sent me a trial version, but it was very unstable. From conversations with him it seems that he is very open to collaborations. However, I am not sure you would enjoy finding youself reverse engineering his code to furnish a prototype, just to discover that web application needs a entirely different set of widgets and programming techniques.

Konstantin

simulation of decoupling

Posted in NMR generator, NMR simulator with tags , , , on January 4, 2009 by kpervushin

I can write a function that would project all interactions of a specified type out of the Liouvillian and put them back when necessary. Something like

[L,C]=decouple(L,spin1,spin2);

… running with ‘decoupled’ L

L=recouple(L,C);

Would that be good enough?

Dear Illya,

a nice suggestion, but does not seem like a solution. The actual decoupling is performed with a sequence of (phase, amplitude modulated) finite pulses run in parallel with all other events on the other remaining channels. Of course, proper simulation of the effect of such a perturbation would require numeric propagation of the density operator through such a complex sequence. If decoupling sequence is suboptimal , it might inflict severe losses of sensitivity. Therefore, adequate simulation of decoupling is an important issue. In fact, in solid state NMR decoupling is an essence of everything with just a slightest misset of power resulting in no signal at all.

I do not see any simple solution at the moment, since this problem belongs to “musician book” (parallel events) category. At some point we have to make an object oriented formal representation of NMR experiment taking care of such an intrinsic parallelism. What bothers me now really is that in the manuscript we have to indicate that composite decoupling is not implemented or implemented as just continuous irradiation droping modulation part whatsoever. This is not devastating but not nice either.

Generally, we do not have an effective (and universal) language describing multi channel sequence of parallel events. I am sure you have seen my inputs to Spinach, they do not look even 1/2 way attractive or elegant to an external user.

Dear Konstantin,

I am in the process thinking about the human-readable input style for Spinach (something along the lines of Gaussian or GAMESS inputs, a stub enclosed – your comments would be appreciated), for both the spin system and the experiment to be run. Gareth Charnock in my group is developing a user-friendly GUI for Spinach (currently a GaussView-style point-and-click spin system specification aid).

We’ll have to think long and hard about pulse sequence specification language. It looks like the most straightforward way would be to implement a “compiler” — a function that would read the user sequence input and generate a stack of Liouvillians through which the spin system has to be propagated (the time grid would probably need to be non-uniform). The propagation as such then would be a straightforward loop.

I’ll look at the way Bruker, Varian, Simpson, Spinevolution etc implement their sequence language and try putting together something neat and flexible. Meanwhile, having a neat spin system specification would be a start – please take a look at the enclosed file. I will write a read_ss.m function that would read it.

While that’s in progress, the “algebraic” decoupling described below should serve as a crutch.

Best wishes,

Ilya.

Dear Ilya,

> I’ll look at the way Bruker, Varian, Simpson, Spinevolution etc implement their sequence language and try putting together something neat and flexible. Meanwhile, having a neat spin system specification would be a start – please take a look at the enclosed file. I will write a read_ss.m function that would read it.

The tagged XML-like description appears to be the natural representation of spin system data. I’d suggest to render it is a class with formal properties, states and associated methods built in this class. This will be totally in style with object oriented programming, reusable and flexible. The actual rendering (point-and-click or anything else) can vary, depending on the platform. This brings the question of the general use of the system.

Strategically I suggest:

1. To make Spinach a server side application.

2. To use ASP.NET with C# as implementation tools. This requires porting Spinach algebraic core to C#, which should be not that difficult to achieve.

Advantages: 1) drastically lowering barrier of the use of the simulator since no local installations are needed just web access. 2) we can use powerful GUI of the web browser to setup analysis and representation of experiments. 3) We can use .NET infrastructure for object oriented programming, access to databases, commonly accepted server side GUI. 4) We add a crucial enhancement to the system in the form of social networked community of users. We can have a public database of experiments, results, documentation, a wiki as well as private sub-collections in development which can be easily exchanged.

NMR generator on ASP.NET

Posted in NMR generator on January 1, 2009 by kpervushin

Google SVN wiki link

http://code.google.com/p/nmrgenerator/wiki/NewSetup?ts=1229446245&updated=NewSetup

Introduction

All the tools are free of charge.

What you will need

  1. .NET Framework 3.5 SP1 and ASP.NET 2.0 (http://msdn.microsoft.com/en-us/netframework/aa569263.aspx)
  2. Visual Web Developer 2008 Express Edition (http://www.microsoft.com/express/vwd/)
  3. TortoiseSVN (http://tortoisesvn.tigris.org/)
  4. MySQL (http://www.mysql.com/) or MSSQL 2008 Express (http://www.microsoft.com/express/sql/default.aspx) – second will be used at the end
  5. MySql Gui Tools (http://dev.mysql.com/downloads/gui-tools/5.0.html)
  6. MySQL ODBC connector (http://dev.mysql.com/downloads/connector/odbc/5.1.html)
  7. MySQL .NET Connector (http://dev.mysql.com/downloads/connector/net/5.2.html)
  8. ILMerge (http://www.microsoft.com/downloads/details.aspx?familyid=22914587-b4ad-4eae-87cf-b14ae6a939b0&displaylang=en)
  9. Straight hands

What you will do

  1. Install .NET framework. ASP.NET will be installed together with it
  2. Install Visual Web Developer 2008
  3. Install TortoiseSVN. This will add additional content to the right click menu.
  4. Find the default folder for your VWD projects. Default is \Documents\Visual Studio 2008\Projects. Right click the folder and do SVN Checkout. In the new window: URL of repository – https://nmrgenerator.googlecode.com/svn/trunk/ nmrgenerator; Checkout Depth – Fully recursive; revision – HEAD revision. When you click ok, you will be asked for credentials to access the repository. Use you gmail account username (the gmail email) you password you will find on this page (http://code.google.com/hosting/settings).
  5. Install MySql, when asked specify root password
  6. Install MySQL GUI Tools
  7. Install All MySQL Connectors
  8. Run ODBC Data Source Administrator – there setup two new DSNs. 1 – Data Source Name = “NMR Experiments”, Server = “localhost”; Port=”3306″ (or the one u set when installed MySQL); user = “nmr”; password = “v rot mne nogi”; Database =”nmrexperiments”. 2 – Data Source Name = “NMR UserData”; Database = “nmruserdata” , the rest is the same.
  9. In the NMR generator project folder locate the folder MySQLDatabase. It contains a backup of DB.
  10. Run “MySQL Administrator” (component of mysql gui tools) with root credentials. In restore tab initiate db restore, when asked select the file from previously located folder. After process is finished the must be three new databases shown in Catalogs tab. In users tab create new user: name – nmr, password – “v rot mne nogi”. Specify SELECT and INSERT permisions for nmrexperiments database and SELECT,INSERT, DELETE, REPLACE and UPDATE for nmruserdata database.
  11. Install ILMerge into default directory
  12. If things work, don’t touch anything, otherwise write here.

New NMR language

Posted in NMR generator, NMR simulator on December 16, 2008 by kpervushin

With the graphs, the only possible issue is that not everyone would have the Bioinformatics Toolbox — in fact, most people that run Matlab would not. A solution would be to copy that function from Bioinformatics Toolbox into the Spinach code tree and adapt it more closely to the problem at hand:

This will be indeed a solution. I will try to implement it. I find graphs to be particularly usefull and tangible in understanding of the experiment.

1. One idea would be to set the normalized coefficient amplitude as a transparency and line thickness parameter and the coefficient phase as a colour — in that way the prominent paths would be visually identifiable without the need to analyze the numbers printed around.

I am not sure transparency can be controlled in biograph object. The line thickness may render the figure messy, but of course one has o try beforehand.


2. Another would be to create plots similar to the traditional coherence order diagrams, with lines jumping between different orders of coherence.

3. A third would be to order the states by their coherence order, assign each state a line of pixels in the picture and generate something resembling a running Fourier transform in acoustic analysis — waves of color would be moving between those lines as the dynamics proceeds. Subspace grouping (see below) would help there.

In fact that was my original idea. However, this would require a substantial effort in programming while biograph is out of box solution. We can ticket it for the future.

The most pressing is parallel programming of events, since without it it seems to be very tricky to correctly model real NMR experiments. Maybe indeed, each event (pulse or delay) might be snapped to the fixed time grid on multiple channels. To calculate real propagators we need to sum up events at this time point from all channels.
I have here in Biozentrum a smart postdoc working on the implementation of the NMR experiments database as the web service (along Web2.0 lines, with the hope to build community of user/developers). We soon make a prototype public. There of course a gateway to Spinach will be implemented. We pondered about comming up with a new syntax for description of NMR experiments like an object model with methods and attributes associated with each class like for example pulse with attributes of length, rotations, shape, point of synchrinization with time grid and methods like relate to calibration data. These classes can be assembled in parallel time lines with a possibility to overlay with complementray methods to resolve overlays. My impression is that this new languge for NMR experiments is bare nessessity. This will make NMR experiments portable across spectrometers and simulators as well.



The similarity metric is a problem. If we define similarity as a scalar product of the two trajectories, then it’s way too strict — a simple phase twist or a half-wave delay would make the two trajectories look completely dissimilar, whereas in reality they are very similar. So…


This brings us very close to analysis of trajectory optimality under given controls. This is a fundamental problem of course. I agree that my metric might be too strict, however, it might be still suitble of we analyse just effects of relaxation keeping all other controls identical for two analyzed trajectories. More thought though should be given..

1. For a given state rho, all its Lz-neighborhood (all states in the exp(-i*Lz*t)*rho orbit) should be deemed equivalent. In practical terms that means that certain coefficients would have to be grouped before taking a scalar product. This deals with the phase issue — if the trajectory passes through a spin in Lx polarization, another trajectory where the same happens in Ly polarization should reasonably be considered similar. In other words, similarity would mean that the trajectory passes through the same subspaces rather than through exactly the same states.

2. The metric should be tolerant to reasonable mismatch of event timings. I do not currently know of any way to implement this robustly (variable Gaussian smoothing? covariation analysis? temporal invariants?). Matthew (who has recently started as a postdoc in Durham) might have some insight into this from his diffractometry and functional analysis background.

3. The metric should be implemented in the Hilbert space sense — as an integral rather than vector scalar product, since the two trajectories might have a different number of points. Interpolation and resampling blues.

4. It might be an idea to only use a small subspace of the full state space for similarity monitoring — in other words, we may want to match up the dynamics of LzSz between the two trajectories, but would not particularly worry about mismatch in the dynamics of third order coherences.

Anyway, a tool created to the above spec (and I can program all four reasonably quickly) would assume quite a lot about the level of user expertise. There might be ways of “black-boxifying” it, but we’d have to make the code proportionally more sophisticated then.


We should first clearly define the problem which we agoing tackle. Say, how different is the actual trajectory from optimal tube of trajectories? This is a very different from the problem of whether relaxation drives the trajectory away besides just attenuating of the magentization. Would be really nice to say something about optimality of experiments.

Time snapping of NMR events

Posted in NMR generator, NMR simulator with tags on December 16, 2008 by kpervushin

Time grid and snapping to time grid will require a new formal description of NMR experiments. New language?

Follow

Get every new post delivered to your Inbox.