Posted: Dec 14, 2020 @ 01:35 PM             Msg. 1 of 33

I am having an issue with the order modification response time taking to long to come back to the application. When we are in a fast moving market and have a trailing stop that moves closer each tick, it falls behind. I have tested this same thing with OEC COM API and GF COM API and the OEC COM API is keeping up with the market while the GF COM API falls behind.

We are updating the orders in the same location as we did for the OEC COM API: OnCommandUpdated and it is taking longer to receive a response to that callback than it does for the OEC COM API.

We are using the client runner to loop through the heartbeat and I don't know if that is just going too slow or what is going on there.

This is best tested on something like the NASDAQ during the morning market when things are moving.

Here is an example of the code we use to modify an order:

IModifyOrderDraftBuilderPtr draftBuilder; draftBuilder.CreateInstance(__uuidof(ModifyOrderDraftBuilder));
draftBuilder->FromOrder(order);
draftBuilder->WithQuantity(qty);
draftBuilder->WithPrice(price);
draftBuilder->WithOrderType(OrderType_Stop);
if (IsValidModification(draftBuilder, order->Contract,m_LocalOrderMap.end()))
{
IModifyOrderDraftPtr ordDraft = draftBuilder->Build(); IOrderPtr order = GFApi()->Orders->ModifyOrder(ordDraft);
if (order)
{
drawnOrder.m_pOrder->SetBeingModified(true);//start modify orderStopWasChanged = true;
}
}

Here is the validation check I am doing:

IsValidModification(IModifyOrderDraftBuilderPtr draftBuilder, GF_Api_COM::IContractPtr contract) { IModifyOrderDraftPtr draft = draftBuilder->Build();
IStringListPtr stringList = GFApi()->Orders->Drafts->Validate(draft);
if (stringList->Count > 0)
return false
return true
}

Posted: Dec 14, 2020 @ 01:36 PM             Msg. 2 of 33

This happens on sim.gainfutures.com, api.gainfutures.com, and prod.gainfutures.com

Posted: Dec 16, 2020 @ 01:42 PM             Msg. 3 of 33

First question is why aren't you using our trailing stop functionality?

Posted: Dec 17, 2020 @ 01:32 PM             Msg. 4 of 33

We allow users to trail off indicators, allow users to jump and then trail, allow users to jump, trail, jump again, and then trail again.

Posted: Dec 22, 2020 @ 03:07 PM             Msg. 5 of 33

I've yet to be able to find anything in particular That should cause order commands to get processed slowly through the COM version of the api.

The COM version of the GF api is just a wrapper around the .NET api, so the speed should be about the same. The only known performance-related issue with the GF API was when dealing with large (100+ account) allocation blocks. A fix has been released for that this weekend (but I need to double check that the fix was pushed to the COM version and nuget)

Please let me know if your testing/development is involving large block orders.

Other than that.. The only thing I can think of that would be causing slow events is something wrong with your event handlers. You mentioned in another thread that you were not getting login-related events at all, did you find what was causing that?

Posted: Dec 22, 2020 @ 06:31 PM             Msg. 6 of 33

My testing/development is not involving large block orders unfortunately.

I was not able to find out what was causing my login issues it just started working again. I'm not going to worry about that for now unless it happens again and I will post in the forum again if I see something happen.


I tested trailing again today putting a timestamp of when I sent off the modification to when I received the modification through OnCommandUpdated callback and I have some timestamps for you. I was wondering if you can look and see on your end if there was something. All of these timestamps are in MST but they are all within about a minute of each other.

I have sectioned out the ones that took >= 3 seconds to come back from the server. This was on the NASDAQ 100, micro-mini March 2021 contract. It was on a 1 minute chart and this one minute bar had a lot of volume.

---- 5 seconds to come back ----

Modifying order with price: 12716.8 at timestamp: 12/22/2020 5:20:23 PM
Modified order coming back with price: 12763 at timestamp: 12/22/2020 5:20:28 PM
Modified order coming back with price: 12716.8 at timestamp: 12/22/2020 5:20:28 PM

----

---- 7 seconds to come back ----

Modifying order with price: 12716.5 at timestamp: 12/22/2020 5:20:29 PM
Modified order coming back with price: 12716.8 at timestamp: 12/22/2020 5:20:34 PM
Modified order coming back with price: 12716.5 at timestamp: 12/22/2020 5:20:36 PM

----

---- 9 seconds to come back ----

Modifying order with price: 12716.3 at timestamp: 12/22/2020 5:20:36 PM
Modified order coming back with price: 12716.5 at timestamp: 12/22/2020 5:20:44 PM
Modified order coming back with price: 12716.3 at timestamp: 12/22/2020 5:20:45 PM

----

---- 6 seconds to come back ----

Modifying order with price: 12716 at timestamp: 12/22/2020 5:20:45 PM
Modified order coming back with price: 12716.3 at timestamp: 12/22/2020 5:20:51 PM
Modified order coming back with price: 12716 at timestamp: 12/22/2020 5:20:51 PM

----

---- 6 seconds to come back ----

Modifying order with price: 12715.8 at timestamp: 12/22/2020 5:20:51 PM
Modified order coming back with price: 12716 at timestamp: 12/22/2020 5:20:56 PM
Modified order coming back with price: 12715.8 at timestamp: 12/22/2020 5:20:57 PM

----

---- 4 seconds to come back ----

Modifying order with price: 12715.5 at timestamp: 12/22/2020 5:20:57 PM
Modified order coming back with price: 12715.8 at timestamp: 12/22/2020 5:21:01 PM
Modified order coming back with price: 12715.5 at timestamp: 12/22/2020 5:21:01 PM

----

---- 3 seconds to come back ----

Modifying order with price: 12715.3 at timestamp: 12/22/2020 5:21:01 PM
Modified order coming back with price: 12715.5 at timestamp: 12/22/2020 5:21:04 PM
Modified order coming back with price: 12715.3 at timestamp: 12/22/2020 5:21:04 PM

----


---- Here is when things started to catch up and trail correctly all taking < 1 second - 1 second to come back ----

Modifying order with price: 12715 at timestamp: 12/22/2020 5:21:04 PM
Modified order coming back with price: 12715.3 at timestamp: 12/22/2020 5:21:05 PM
Modified order coming back with price: 12715 at timestamp: 12/22/2020 5:21:05 PM
Modifying order with price: 12714.8 at timestamp: 12/22/2020 5:21:05 PM
Modified order coming back with price: 12715 at timestamp: 12/22/2020 5:21:05 PM
Modified order coming back with price: 12714.8 at timestamp: 12/22/2020 5:21:06 PM
Modifying order with price: 12714.5 at timestamp: 12/22/2020 5:21:06 PM
Modified order coming back with price: 12714.8 at timestamp: 12/22/2020 5:21:06 PM
Modified order coming back with price: 12714.5 at timestamp: 12/22/2020 5:21:06 PM
Modifying order with price: 12714.3 at timestamp: 12/22/2020 5:21:07 PM
Modified order coming back with price: 12714.5 at timestamp: 12/22/2020 5:21:07 PM
Modified order coming back with price: 12714.3 at timestamp: 12/22/2020 5:21:07 PM
Modifying order with price: 12714 at timestamp: 12/22/2020 5:21:07 PM
Modified order coming back with price: 12714.3 at timestamp: 12/22/2020 5:21:07 PM
Modified order coming back with price: 12714 at timestamp: 12/22/2020 5:21:08 PM
Modifying order with price: 12713.8 at timestamp: 12/22/2020 5:21:08 PM
Modified order coming back with price: 12714 at timestamp: 12/22/2020 5:21:08 PM
Modified order coming back with price: 12713.8 at timestamp: 12/22/2020 5:21:08 PM
Modifying order with price: 12713.5 at timestamp: 12/22/2020 5:21:09 PM
Modified order coming back with price: 12713.8 at timestamp: 12/22/2020 5:21:09 PM
Modified order coming back with price: 12713.5 at timestamp: 12/22/2020 5:21:09 PM
Modifying order with price: 12713.3 at timestamp: 12/22/2020 5:21:09 PM
Modified order coming back with price: 12713.5 at timestamp: 12/22/2020 5:21:09 PM
Modified order coming back with price: 12713.3 at timestamp: 12/22/2020 5:21:10 PM
Modifying order with price: 12713 at timestamp: 12/22/2020 5:21:10 PM
Modified order coming back with price: 12713.3 at timestamp: 12/22/2020 5:21:10 PM
Modified order coming back with price: 12713 at timestamp: 12/22/2020 5:21:10 PM
Modifying order with price: 12712.8 at timestamp: 12/22/2020 5:21:11 PM
Modified order coming back with price: 12713 at timestamp: 12/22/2020 5:21:11 PM
Modified order coming back with price: 12712.8 at timestamp: 12/22/2020 5:21:11 PM
Modifying order with price: 12712.5 at timestamp: 12/22/2020 5:21:11 PM
Modified order coming back with price: 12712.8 at timestamp: 12/22/2020 5:21:11 PM
Modified order coming back with price: 12712.5 at timestamp: 12/22/2020 5:21:12 PM
Modifying order with price: 12712.3 at timestamp: 12/22/2020 5:21:12 PM
Modified order coming back with price: 12712.5 at timestamp: 12/22/2020 5:21:12 PM
Modified order coming back with price: 12712.3 at timestamp: 12/22/2020 5:21:12 PM
Modifying order with price: 12712 at timestamp: 12/22/2020 5:21:12 PM
Modified order coming back with price: 12712.3 at timestamp: 12/22/2020 5:21:12 PM
Modified order coming back with price: 12712 at timestamp: 12/22/2020 5:21:12 PM
Modifying order with price: 12711.8 at timestamp: 12/22/2020 5:21:13 PM
Modified order coming back with price: 12712 at timestamp: 12/22/2020 5:21:13 PM
Modified order coming back with price: 12711.8 at timestamp: 12/22/2020 5:21:13 PM
Modifying order with price: 12711.5 at timestamp: 12/22/2020 5:21:13 PM
Modified order coming back with price: 12711.8 at timestamp: 12/22/2020 5:21:13 PM
Modified order coming back with price: 12711.5 at timestamp: 12/22/2020 5:21:14 PM
Modifying order with price: 12711.3 at timestamp: 12/22/2020 5:21:14 PM
Modified order coming back with price: 12711.5 at timestamp: 12/22/2020 5:21:14 PM
Modified order coming back with price: 12711.3 at timestamp: 12/22/2020 5:21:14 PM
Modifying order with price: 12711 at timestamp: 12/22/2020 5:21:14 PM
Modified order coming back with price: 12711.3 at timestamp: 12/22/2020 5:21:15 PM
Modified order coming back with price: 12711 at timestamp: 12/22/2020 5:21:15 PM
Modifying order with price: 12710.8 at timestamp: 12/22/2020 5:21:15 PM
Modified order coming back with price: 12711 at timestamp: 12/22/2020 5:21:15 PM
Modified order coming back with price: 12710.8 at timestamp: 12/22/2020 5:21:15 PM
Modifying order with price: 12710.5 at timestamp: 12/22/2020 5:21:15 PM
Modified order coming back with price: 12710.8 at timestamp: 12/22/2020 5:21:16 PM
Modified order coming back with price: 12710.5 at timestamp: 12/22/2020 5:21:16 PM
Modifying order with price: 12710.3 at timestamp: 12/22/2020 5:21:16 PM
Modified order coming back with price: 12710.5 at timestamp: 12/22/2020 5:21:16 PM
Modified order coming back with price: 12710.3 at timestamp: 12/22/2020 5:21:17 PM
Modifying order with price: 12710 at timestamp: 12/22/2020 5:21:17 PM
Modified order coming back with price: 12710.3 at timestamp: 12/22/2020 5:21:17 PM
Modified order coming back with price: 12710 at timestamp: 12/22/2020 5:21:17 PM
Modifying order with price: 12709.8 at timestamp: 12/22/2020 5:21:18 PM
Modified order coming back with price: 12710 at timestamp: 12/22/2020 5:21:18 PM
Modified order coming back with price: 12709.8 at timestamp: 12/22/2020 5:21:18 PM
Modifying order with price: 12709.5 at timestamp: 12/22/2020 5:21:18 PM
Modified order coming back with price: 12709.8 at timestamp: 12/22/2020 5:21:19 PM
Modified order coming back with price: 12709.5 at timestamp: 12/22/2020 5:21:19 PM
Modifying order with price: 12709.3 at timestamp: 12/22/2020 5:21:19 PM
Modified order coming back with price: 12709.5 at timestamp: 12/22/2020 5:21:19 PM
Modified order coming back with price: 12709.3 at timestamp: 12/22/2020 5:21:19 PM
Modifying order with price: 12709 at timestamp: 12/22/2020 5:21:19 PM
Modified order coming back with price: 12709.3 at timestamp: 12/22/2020 5:21:20 PM
Modified order coming back with price: 12709 at timestamp: 12/22/2020 5:21:20 PM
Modifying order with price: 12708.8 at timestamp: 12/22/2020 5:21:20 PM
Modified order coming back with price: 12709 at timestamp: 12/22/2020 5:21:20 PM
Modified order coming back with price: 12708.8 at timestamp: 12/22/2020 5:21:20 PM

Posted: Dec 23, 2020 @ 06:36 AM             Msg. 7 of 33

Can I have the order ID(s) so I can dig through our logs?

Posted: Dec 23, 2020 @ 10:07 AM             Msg. 8 of 33

And I also want to confirm that you are creating an instance to the API just once, and creating a runner just once.

i.e: (c++)

GF_Api_COM::IGFComClientPtr _GFAPI;
GF_Api_COM::IGFComClientRunnerPtr _GFClientRunner;

_GFAPI.CreateInstance(__uuidof(GF_Api_COM::GFComClient));
_GFClientRunner = _GFAPI->Threading->CreateRunnerFor(_GFAPI);
_GFClientRunner->Start();

Posted: Dec 23, 2020 @ 11:29 AM             Msg. 9 of 33

Sorry I didn't get the order ID's. Let me duplicate it again and get some order ID's for you. I'm not sure if you can just search by a username/id but there should be only one order opened at that time under my username.

I can confirm I am creating an instance to the API just once and creating a runner just once.

I actually had a question about that. I was wondering if there is a possibility that due to the number of price ticks coming in and doing an order modification shortly after it comes in, is there a way that the message queue gets clogged up and price ticks take precedence over an order modification in the message queue?

I know there are a few different servers you can login to: Order, Price, and Aggregate so I was wondering if it is a possibility to login to the price server and order server using two separate client runners so two threads are working at the same time.

Posted: Dec 23, 2020 @ 11:43 AM             Msg. 10 of 33

I was able to duplicate it again. Here is the order ID: 211823182

---- 11 seconds ----
Modifying order with price: 12703.8 at timestamp: 12/23/2020 10:39:01 AM
Modified order id: 211823182 coming back with price: 12657.5 at timestamp: 12/23/2020 10:39:06 AM
Modified order id: 211823182 coming back with price: 12703.8 at timestamp: 12/23/2020 10:39:12 AM

---- 10 seconds ----
Modifying order with price: 12704 at timestamp: 12/23/2020 10:39:12 AM
Modified order id: 211823182 coming back with price: 12703.8 at timestamp: 12/23/2020 10:39:15 AM
Modified order id: 211823182 coming back with price: 12704 at timestamp: 12/23/2020 10:39:22 AM

---- 12 seconds ----
Modifying order with price: 12704.3 at timestamp: 12/23/2020 10:39:27 AM
Modified order id: 211823182 coming back with price: 12704 at timestamp: 12/23/2020 10:39:32 AM
Modified order id: 211823182 coming back with price: 12704.3 at timestamp: 12/23/2020 10:39:39 AM

---- 10 seconds ----
Modifying order with price: 12704.5 at timestamp: 12/23/2020 10:39:39 AM
Modified order id: 211823182 coming back with price: 12704.3 at timestamp: 12/23/2020 10:39:45 AM
Modified order id: 211823182 coming back with price: 12704.5 at timestamp: 12/23/2020 10:39:49 AM

---- 12 seconds ----
Modifying order with price: 12704.8 at timestamp: 12/23/2020 10:39:50 AM
Modified order id: 211823182 coming back with price: 12704.5 at timestamp: 12/23/2020 10:39:55 AM
Modified order id: 211823182 coming back with price: 12704.8 at timestamp: 12/23/2020 10:40:02 AM

---- 10 seconds ----
Modifying order with price: 12705 at timestamp: 12/23/2020 10:40:03 AM
Modified order id: 211823182 coming back with price: 12704.8 at timestamp: 12/23/2020 10:40:08 AM
Modified order id: 211823182 coming back with price: 12705 at timestamp: 12/23/2020 10:40:13 AM

Posted: Dec 23, 2020 @ 12:19 PM             Msg. 11 of 33

The aggregate connection is just an abstraction around a order + price connection for ease of use.
A second runner really wouldn't do anything, as it just calls GFApi.Threading.Advance in a loop ever 10 ms.

The COM version of both the old OEC and current GF apis were just wrappers around the .NET api. I'm digging into differences in implementation of the wrappers but I'm not really seeing much.

I'll look through the server logs for that order ID now, and let you know what i've found.

Posted: Dec 23, 2020 @ 02:01 PM             Msg. 12 of 33

Ok, this is really weird..

On the server that users log into for orders, the first modify (to 12703.75) took ~4 seconds from being received from your app to being fully modified.

But if i look at the logs for where the order is actually processed... it received the modify command, checked risk, and had the order fully modified (and responses transmitted) in.. 84ms.

There must be something off with the API environment, as we haven't had complaints of order delays this bad anywhere else.

Posted: Dec 23, 2020 @ 02:15 PM             Msg. 13 of 33

Are you suggesting it's something on our end?

If so:
1. Do you have any suggestions on how I can better diagnose/narrow down what is going on
2. How do you explain the ~4 second time your server is logging if that doesn't have anything to do with our application since it is from the time of receiving it and modifying it?

Thanks for looking into it and all of your help.

Posted: Dec 23, 2020 @ 02:41 PM             Msg. 14 of 33

It may be something on our end. I'll hopefully have some answers soon.

Posted: Dec 23, 2020 @ 02:42 PM             Msg. 15 of 33

I appreciate you, thank you.

Posted: Dec 30, 2020 @ 10:13 PM             Msg. 16 of 33

Hi Jason, I'm wondering if there has been any forward progress on this issue?

Thanks

Posted: Jan 04, 2021 @ 07:33 AM             Msg. 17 of 33

Sorry, I was on vacation between Christmas and New Years. Back at it today.

Before I left I did see one possibility of the source, but no confirmation.

a few questions:
Does this issue happen any time of the day on API? I'd like to know if it still happens after ~1pm CST
Does this issue occur on SIM environment.. if so, can you send me a order ID example?

Posted: Jan 04, 2021 @ 12:53 PM             Msg. 18 of 33

This issue happens at any time of the day. I have seen it happen after ~1 pm CST.

This issue does occur on sim, api, and prod. Let me try and duplicate on sim and get you an order id.

Thanks

Posted: Jan 04, 2021 @ 01:13 PM             Msg. 19 of 33

In sim.gainfutures.com - Order ID 251533343

--- 54 seconds ---
Modifying order id: 251533343 with price: 12657.5 at timestamp: 1/4/2021 12:08:52 PM
Modified order id: 251533343 coming back with price: 12703.8 at timestamp: 1/4/2021 12:09:46 PM
Modified order id: 251533343 coming back with price: 12657.5 at timestamp: 1/4/2021 12:10:00 PM

--- 54 seconds ---
Modifying order id: 251533343 with price: 12657.3 at timestamp: 1/4/2021 12:10:00 PM
Modified order id: 251533343 coming back with price: 12657.5 at timestamp: 1/4/2021 12:10:53 PM
Modified order id: 251533343 coming back with price: 12657.3 at timestamp: 1/4/2021 12:10:54 PM

--- <= 1 second (working as intended) ---
Modifying order id: 251533343 with price: 12657 at timestamp: 1/4/2021 12:10:54 PM
Modified order id: 251533343 coming back with price: 12657.3 at timestamp: 1/4/2021 12:10:54 PM
Modified order id: 251533343 coming back with price: 12657 at timestamp: 1/4/2021 12:10:54 PM
Modifying order id: 251533343 with price: 12656.8 at timestamp: 1/4/2021 12:10:54 PM
Modified order id: 251533343 coming back with price: 12657 at timestamp: 1/4/2021 12:10:55 PM
Modified order id: 251533343 coming back with price: 12656.8 at timestamp: 1/4/2021 12:10:55 PM
Modifying order id: 251533343 with price: 12656.5 at timestamp: 1/4/2021 12:10:55 PM
Modified order id: 251533343 coming back with price: 12656.8 at timestamp: 1/4/2021 12:10:55 PM
Modified order id: 251533343 coming back with price: 12656.5 at timestamp: 1/4/2021 12:10:56 PM
Modifying order id: 251533343 with price: 12656.3 at timestamp: 1/4/2021 12:10:56 PM
Modified order id: 251533343 coming back with price: 12656.5 at timestamp: 1/4/2021 12:10:56 PM
Modified order id: 251533343 coming back with price: 12656.3 at timestamp: 1/4/2021 12:10:56 PM
Modifying order id: 251533343 with price: 12656 at timestamp: 1/4/2021 12:10:56 PM
Modified order id: 251533343 coming back with price: 12656.3 at timestamp: 1/4/2021 12:10:56 PM
Modified order id: 251533343 coming back with price: 12656 at timestamp: 1/4/2021 12:10:56 PM
Modifying order id: 251533343 with price: 12655.8 at timestamp: 1/4/2021 12:10:57 PM
Modified order id: 251533343 coming back with price: 12656 at timestamp: 1/4/2021 12:10:57 PM
Modified order id: 251533343 coming back with price: 12655.8 at timestamp: 1/4/2021 12:10:58 PM
Modifying order id: 251533343 with price: 12655.5 at timestamp: 1/4/2021 12:10:58 PM
Modified order id: 251533343 coming back with price: 12655.8 at timestamp: 1/4/2021 12:10:58 PM
Modified order id: 251533343 coming back with price: 12655.5 at timestamp: 1/4/2021 12:10:58 PM
Modifying order id: 251533343 with price: 12655.3 at timestamp: 1/4/2021 12:10:59 PM
Modified order id: 251533343 coming back with price: 12655.5 at timestamp: 1/4/2021 12:10:59 PM
Modified order id: 251533343 coming back with price: 12655.3 at timestamp: 1/4/2021 12:10:59 PM

Posted: Jan 04, 2021 @ 01:22 PM             Msg. 20 of 33

If it helps out at all to look at a different path I am seeing slow response times for placing orders too.

Took 15 seconds to receive a response from the server to confirm an order.

In sim.gainfutures.com order ID 251534737
Creating new order at 1/4/2021 12:20:40 PM
Order confirmed at 1/4/2021 12:20:55 PM