Talk:JSON link protocol

Support for bare TCP/UDP packet transfers
(discussion moved from Talk:JSON link --Dchassin 15:49, 11 April 2013 (UTC))

--Bpalmintier 18:53, 9 April 2013 (UTC)

In some situations, it may be undesirable to require an HTTP compliant host on the other end of the JSON link. This is especially true for hardware based hosts such as hardware-in-the-loop testing or simple embedded systems based nodes. Such systems may not want to spin up a full HTTP daemon.

I suggest adding support for putting the JSON data objects directly into UDP or TCP packets. Such simple sockets are easy to implement and have lower host overhead than HTTP. Furthermore in many multi-machied time synced model scenarios, the "guaranteed" delivery of TCP (which also underlies HTTP) could result in attempting to re-transmit old, no longer valid data in the event of network troubles. In such scenarios, direct UDP packets can be appropriate. In fact, real-time applications are often one of the key advantages listed for UDP packets. E.g. see Wikipedia's UDP entry.

Recommend adding a "protocol" command to GLM (and/or link control file), including the following options:
 * HTTP: use HyperText Transfer Protocol (HTTP) for use with "web" servers
 * TCP: Exchange data using Transmission Control Protocol (TCP) packets directly. TCP still provides congestion control and reliable delivery at the possible expense of data synchronization issues with network troubles
 * UDP: Exchange data with User Datagram Protocol (UDP) packets directly. UDP packets are only sent once with no guarantees, but such service may be desirable for fast real-time simulations

--Dchassin 19:44, 9 April 2013 (UTC)

In the case of TCP and UDP, a packet format must be specified to allow transmission of additional necessary information that normally comes in the HTTP header and is required for proper operation. Specifically:
 * header size (number of bytes to beginning of payload)
 * error status (e.g., OK, ERROR)
 * message length (number of bytes in payload)
 * connection status (e.g., ALIVE, CLOSED)

--Bpalmintier 20:03, 9 April 2013 (UTC)

Much of this information is already included in UDP and TCP packet headers. Specifically: For more information see: [UDP packet structure] and [TCP packet structure]
 * header size is predefined
 * message length is included in the header
 * a checksum for simple data integrity checks

Error/connection status are not specified, although some of this is generally already handled by the standard socket libraries in most languages. At the application layer (ie GridLAB-D JSON data exchange) the error status, etc. would be handled using some form of the JSON link protocol defined by GridLAB-D.

--Dchassin 20:07, 9 April 2013 (UTC)

I'm no expert on UDP and TCP, but it's my understanding that the packet header in UDP and TCP is for that single packet, not the entire message payload. I don't believe applications are expected to overload the meaning of that header.

--Dchassin 20:12, 9 April 2013 (UTC)

Regarding a new protocol directive, that is not necessary because the protocol is already part of the port directive. We should simply add HTTP to it.

--Bpalmintier 23:28, 9 April 2013 (UTC)

RE: packet metadata:

I am thinking that for many uses the entire data/object will fit into a single packet. In this case there is no need to specify any additional size information. For most of our near-mid term hardware usage this should be sufficient, and could even be a limitation of for the TCP and UDP interchanges. In practice with my TCP/UDP based JSON test script I am able to reliably send & receive over 300 double values with 2-4 character keys over UDP and about 100 double values with 2-4 character keys over TCP. Both of these limits are totally fine for many applications. And if more points are needed, they could be accommodated by having multiple links going in parallel.

Re: protocol directive:

I think the key is that the port and protocol are somehow specified independently. This could be done in a single directive or separately. For example simply because HTTP normally uses port 80, it is still possible (though not recommended) to receive bare TCP/UDP on port 80, and it is very possible to use HTTP over a different port entirely. So it is useful to be able to specify e.g. port=2134 protocol=UDP wether its in a single directive or more.

--Dchassin 17:53, 10 April 2013 (UTC)

The packet structure will have its own header. We will not overload the UDP/TCP header with metadata from the payload. That is very bad practice.

The port and protocol are already separate, e.g., "HTTP/80" or "TCP/39035" or "UDP/50007" but provided in the same directive.

--Dchassin 16:20, 11 April 2013 (UTC)

See Raw TCP/UDP section for details.

--Dchassin 16:28, 11 April 2013 (UTC)

One issue that may come up is the UDP does not provide payload integrity checks. This means we should consider having a checksum in the header to help the application verify that the data has not been altered in transit. The current proposed header format for TCP/UDP do not provide for that.

--Bpalmintier 23:53, 11 April 2013 (UTC) This seems like a great start and I look forward to working to refine the protocol going forward. A few initial thoughts:

For the UDP application especially I think it is important to minimize any added overhead/processing needs. As a result I wonder if we really need all of the proposed header fields and if some of them can be shortened. Some specific questions/suggestions:
 * What is the function of the response code? It looks like its trying to duplicate some HTTP functionality, but in this case do we need that? Seems like either the packet makes it or it doesn't and that the higher level JSON based information would describe other errors
 * The same thing is true for the connection status. This seems unnecessary
 * For TCP I see the utility of defining a data length since TCP might opt to arbitrarily segment the data so its nice to know how much is coming. I am still not convinced this is needed for UDP, since the use case is to fit everything in a single packet. If the packet makes it we read/process it all, if it doesn't then no data. And if only part makes it it should be obvious since the JSON will be no good. ie no terminating }
 * Can we streamline the numeric values to binary? If we are working in byte offsets perhaps we don't need ascii versions of numbers, or even space padding for that matter

For the JSON meta data:
 * How does the msg-num get used? Is it redundant if we are also providing the timestamp?
 * When sending the input/output schemas, perhaps the method should say "in_schema" and "out_schema", and/or the schema JSON payload could be in_schema/out_schema
 * (minor) I find the "method" name slightly confusing, would "action" work? or maybe one of the shorter options below
 * thinking again about overhead, what do you think of shortening some of the keywords a bit: e.g. "act" or "do" for method, "reply" or "ack" for response, "in" and "out", etc.
 * taking it a step further, it could be that we don't need a method parameter at all. The names for payload sub-object could encode the message type. e.g. app_data={...}, remote_data, start, error, sync, end, etc.
 * It could help to "description" and "version" fields to the remote initialization reply.

I am very excited about UDP for data exchange, with a likely limitation that everything fits into a single packet. But for some of the initial handshaking and especially for any schema exchange it might make sense to start with TCP and then switch to UDP.

For UDP data integrity, the UDP header does include a checksum for both data and header. This is the same level of integrity checks provided by bare TCP, so we probably don't need any more.

--Dchassin 04:32, 12 April 2013 (UTC)

I think we have enough to get the job done now. Let's get the link working and after we've debugged everything you can develop you're own variant (as long as you don't break this one). After tomorrow I'm unavailable for several months, so we'd better get it working fast instead of trying to guild the lily.

--Bpalmintier 18:57, 19 April 2013 (UTC)

I am working through a state diagram for the link protocol and have realized a few more pieces of data to exchange:
 * The refresh interval... which might be used by the remote machine when setting timeouts. The GridLAB machine knows this through the link file, the remote machine does not
 * Some indication of what to do when data is invalid. The Gridlab machine knows this through the link file, but the remote machine does not.

--Dchassin 19:24, 19 April 2013 (UTC)

The timeout is given in the header data (keep-alive time) because timeouts are usually implemented by transport layer. It is not placed in the payload.

What are the remote machine's choices if the data is invalid? I believe there's only one thing to do, and that's ignore the data and wait for another message or for the connection to close.

--Bpalmintier 20:57, 19 April 2013 (UTC)

Refresh Interval: Hmm, I do see the communication timeout now, and I can see some utility in having that feature at the transport level, but what I am asking about is at the application level: how fast should the remote machine expect to get updates from GridLAB during the run? ie what is the data refresh rate. This is different from (though possibly related to) the comm. timeout. Seems like refresh rate (maybe a float in units of seconds? Or an int in ms?) is a field that would be useful to add to the start (or maybe init) exchange.

For the Comm timeout, what are the units? I'm guessing seconds, in which case I think we might want more fine grain information than 1-9. e.g. could be 0.5sec or even 0.05sec for fast/local stuff (incluing delta mode on GridLAB)

I think the options are comparable to GridLAB's options: ABORT, RETRY, IGNORE (ie use old data), SIM (ie use new local data)

I just posted a state diagram for the slave: thoughts? One idea that comes up is if we want a separate STOP and TERM. Stop to simply stop/pause the simulation and TERM to completely exit.

--Dchassin 16:46, 22 April 2013 (UTC)

There is no direct control over the refresh interval except through the run_realtime global variable. If you want to pass that variable through to the remote server, just add it to the global link directive.