Posted: Apr 13, 2021 @ 06:01 PM             Msg. 1 of 43

It appears that the ZSKM21 (Soybeans May 2021 contract) is not returning all of the history for intraday charts if the number of bars exceeds 4096.

This appears to only happen in the sim version

After the first 4096 bars, the history request doesn't come back anymore

Here is an example of a history request we do:
OECAPICOM::IContractPtr con = OECAPI()->Contracts->Item("ZSKM21");
OECAPI()->RequestBars_3(con, 1.9965, 44299.8958, OECAPICOM::SubscriptionType_Bar, 1);

This appeared to have broke over the weekend.

I do realize this is still the OECAPI, we are currently in the process of moving over to the GF API.

Will you please take a look into this?

Thanks

Posted: Apr 13, 2021 @ 07:06 PM             Msg. 2 of 43

Sorry, to clarify, I tested this in the GF API and it is broken in there as well.

Again, this is sim.gainfutures.com.

We are aware of Soybeans and Corn that are broken right now, but they are broken in both the OEC API and the GF API.

Thanks

Posted: Apr 14, 2021 @ 10:22 AM             Msg. 3 of 43

I'm not seeing an issue with 1-minute bars on ZS with our FIX or .NET applications. I'll check with devs to confirm with COM API.

Posted: Apr 14, 2021 @ 10:23 AM             Msg. 4 of 43

Is ZSKM21 a typo? Did you mean ZSK21 or ZSM21?

Posted: Apr 14, 2021 @ 11:33 AM             Msg. 5 of 43

Yes sorry, ZSK21 is what I meant.

Posted: Apr 15, 2021 @ 12:35 PM             Msg. 6 of 43

I just loaded up a GF COM example code, and ran:

GF_Api_COM::IContractPtr contract = GFAPI()->Contracts->Get_2((LPCSTR)dlg.m_SelectedSymbol);
GFAPI()->Subscriptions->Bars->Subscribe(
contract->id,
GFAPI()->Subscriptions->Bars->Duration->Create_3(8000, GF_Api_COM::Continuity_OneTime),
GFAPI()->Subscriptions->Bars->Description->CreateMinutes(1)
);

for ZSK21 on API and SIM, and both worked fine (one message of 4096 bars, one of 3905)

Posted: Apr 16, 2021 @ 12:47 PM             Msg. 7 of 43

Thanks for checking it out. I'll see if I can narrow down the issue and post back here when I have more information.

Posted: Apr 16, 2021 @ 02:14 PM             Msg. 8 of 43

It is random how many times I am able to get a callback.

There are times when no callback occurs and other times when I get 4096 or get up to 16,398 records, but stop after that.

It seems like it might be some type of data lock preventing the callback from coming. This is what I get from a debug diag analysis:

win32u!NtUserMsgWaitForMultipleObjectsEx+c
user32!RealMsgWaitForMultipleObjectsEx+7a
user32!MsgWaitForMultipleObjectsEx+4c
combase!CCliModalLoop::BlockFn+14b [onecore\com\combase\dcomrem\callctrl.cxx @ 2156 + d] onecore\com\combase\dcomrem\callctrl.cxx @ 2156 + d
combase!ClassicSTAThreadWaitForHandles+b4 [onecore\com\combase\dcomrem\classicsta.cpp @ 51 + e] onecore\com\combase\dcomrem\classicsta.cpp @ 51 + e
combase!CoWaitForMultipleHandles+77 [onecore\com\combase\dcomrem\sync.cxx @ 122 + c] onecore\com\combase\dcomrem\sync.cxx @ 122 + c
mshtml!CDwnTaskExec::ThreadExec+102
mshtml!CExecFT::StaticThreadProc+6d
kernel32!BaseThreadInitThunk+19
ntdll_77720000!__RtlUserThreadStart+2f
ntdll_77720000!_RtlUserThreadStart+1b

I started going off the path of thinking there needed to be a closed position of Soybeans for this to happen, but that was not true. I tested with an account that has not had a closed position of soybeans.

I'm not quite sure what else to give you for data as to why I'm not getting a callback.

Posted: Apr 20, 2021 @ 06:49 PM             Msg. 9 of 43

I tested this again today with the same results. Here is an example of what I am doing in the GF API.

Keep in mind, this works for all other symbols so I'm sure that it's not the request that is the issue.

IDurationPtr duration = GFApi()->Subscriptions->Bars->Duration->Create_2(1.9965 44299.8958, GF_Api_COM::Continuity::Continuity_OneTime);
IBarDescriptionPtr description = GFApi()->Subscriptions->Bars->Description->CreateMinutes(1);
GFApi()->Subscriptions->GetBars()->Subscribe(con->id, duration, description);

It works correctly through api.gainfutures.com and through prod.gainfutures.com, but through sim.gainfutures.com, it stops sending back bars after some random amount of records (different every time). I have even had all of the bars come through a couple times as well.

Seems like something gets clogged in the pipeline when sending back the bars and doesn't get unstuck. If you hit it at the right time, things will go through which makes me think it is some sort of mutex blocking the bars from coming back and it never gets unlocked.

Posted: Apr 21, 2021 @ 02:07 PM             Msg. 10 of 43

I have been testing this exact code repeatedly today, on SIM. sometimes as fast as once a second, and never have i seen it not get all the data correctly.

The only thing i can think of is that you are doing some sort of intensive processing in your OnBarsReceived callback that's slowing down the next message processing.

Posted: Apr 21, 2021 @ 02:28 PM             Msg. 11 of 43

My argument for that is that our OnBarsReceived callback code has not changed in a year.

I reverted our code to 2 months ago and we are still seeing the same issue and have only recently had issues with this.

Any other things you can think of I can test on my side to help narrow down this issue?

Thanks

Posted: Apr 23, 2021 @ 09:38 AM             Msg. 12 of 43

The only known bar-related bug that i know about was for about a week on API, close timestamps weren't being filled out, but that issue was fixed before it went to SIM.

I still have yet to be able to replicate the issue, and everything that is currently on API will be released to SIM this weekend, so hopefully this issue will go away.

Posted: Apr 26, 2021 @ 02:10 PM             Msg. 13 of 43

Unfortunately, the api fixes did not fix what is happening for us in sim. I'm struggling to see an issue on our end especially since it works fine in a real money account. Additionally, it works for all other symbols (besides corn from what we have found), so I don't see an issue on our end with our callback.

Our current work-around is that we log into prod and since we are caching the data, we are able to pull in the prod data and then use that cached data in sim. We then request the difference between the cached timestamp to the current time. This works because we aren't requesting as many bars and it is able to finish before something locks up.

This isn't a valid work-around though since all users to don't have a real account and some people only paper trade.

Thank you for your time on this issue, but we are still seeing the issue after this weeks changes posted to sim.

Posted: Apr 27, 2021 @ 06:52 AM             Msg. 14 of 43

Ok. maybe some logging will help me determine what's going on.

here's what i need you to do.
In your code that creates the GFAPI instance, add:

GFAPI()->Logging->SetCategories(GF_Api_COM::LogCategory_All);

Add a handler for OnNewMessageLogged.
Output the current timestamp, category, and message to a file.

Try to get a log for the time when the issues occurs on SIM, and note the timestamp when it was occurring.

Posted: Apr 27, 2021 @ 12:05 PM             Msg. 15 of 43

Here is the logging that happens when requesting the 1 minute data - all times are in MST:

Timestamp: 4/27/2021 11:01:42 AM - Category: 0 - Message: 3 ZSK21: 0 Bars
Timestamp: 4/27/2021 11:01:43 AM - Category: 2 - Message: PriceConnection Outgoing:
Timestamp: 4/27/2021 11:01:46 AM - Category: 0 - Message: 4 ZSK21: 4096 Bars
Timestamp: 4/27/2021 11:01:46 AM - Category: 0 - Message: 4 ZSK21: 4096 Bars
Timestamp: 4/27/2021 11:01:46 AM - Category: 0 - Message: 4 ZSK21: 4096 Bars
Timestamp: 4/27/2021 11:01:46 AM - Category: 0 - Message: 4 ZSK21: 4096 Bars
Timestamp: 4/27/2021 11:01:46 AM - Category: 0 - Message: 4 ZSK21: 4096 Bars
Timestamp: 4/27/2021 11:01:46 AM - Category: 0 - Message: PriceConnection Disconnected: System.IO.IOException: Disconnected

at GF.Net.PacketSocket.ReadLen(Byte[] buf, Int32 length)

at GF.Net.PacketSocket.Read()

at GF.Net.ProtoSocket.Read[T](Func`2 deserialize)

at GF.Net.ProtoClient`2.ReadLoop()
Timestamp: 4/27/2021 11:01:46 AM - Category: 0 - Message: 4 ZSK21: 4096 Bars
Timestamp: 4/27/2021 11:01:46 AM - Category: 0 - Message: 4 ZSK21: 4096 Bars
Timestamp: 4/27/2021 11:01:46 AM - Category: 0 - Message: OrderConnection Disconnected: System.Exception: by request


It looks like I get disconnected from the API which I didn't notice before. The OEC API does not get disconnected or get an error message.
Edited by RWare2020 on Apr 27, 2021 12:05 PM

Posted: Apr 28, 2021 @ 07:24 AM             Msg. 16 of 43

I'm looking into what might be causing that disconnect message.

Posted: Apr 28, 2021 @ 10:27 AM             Msg. 17 of 43

What user name are you using on sim?

Posted: Apr 28, 2021 @ 10:59 AM             Msg. 18 of 43

Thank you.

I am using TNT2

Posted: Apr 28, 2021 @ 11:13 AM             Msg. 19 of 43

Here is another log from today that seems like it got a little bit farther - Again, times are in MST:

Timestamp: 4/28/2021 10:11:26 AM - Category: 2 - Message: PriceConnection Outgoing:
Timestamp: 4/28/2021 10:11:29 AM - Category: 0 - Message: 10 ZSK21: 4096 Bars
Timestamp: 4/28/2021 10:11:29 AM - Category: 0 - Message: 10 ZSK21: 4096 Bars
Timestamp: 4/28/2021 10:11:29 AM - Category: 0 - Message: PriceConnection Disconnected: System.Net.Sockets.SocketException (0x80004005): An existing connection was forcibly closed by the remote host
at System.Net.Sockets.Socket.Receive(Byte[] buffer, Int32 offset, Int32 size, SocketFlags socketFlags)
at GF.Net.GFSocket.Receive(Byte[] buffer, Int32 offset, Int32 size, SocketFlags socketFlags)
at GF.Net.PacketSocket.ReadLen(Byte[] buf, Int32 length)
at GF.Net.PacketSocket.Read()
at GF.Net.ProtoSocket.Read[T](Func`2 deserialize)
at GF.Net.ProtoClient`2.ReadLoop()

Posted: Apr 28, 2021 @ 12:25 PM             Msg. 20 of 43

hmm... i see the reason for the disconnect:

11:11:28.243 WARN [tid AsyncAppender] sid=(null) cid=(null) acc=(null) ord=(null) - Slow connection detected: 50.77.44.92:62120: 17 items, est. ram: 6145 KB
11:11:28.348 WARN [tid AsyncAppender] sid=(null) cid=(null) acc=(null) ord=(null) - Closing slow connection: 50.77.44.92:62120: 30 items, est. ram: 11265 KB
11:11:28.349 INFO [tid AsyncAppender] sid=(null) cid=(null) acc=(null) ord=(null) - Logged off 50.77.44.92:62120:TNT2 Slow connection
(timestamps in CST)

It looks like you're not processing price updates fast enough, so we disconnect you to limit memory usage . In this case there were 11mB of price updates that you hadn't processed.