QuickFIX: Load a message from the logs

QuickFIX: Loading and Replaying Messages from Logs

Learn how to extract, parse, and replay FIX messages from QuickFIX session logs for debugging, testing, and analysis.

QuickFIX is a robust open-source FIX engine that handles the complexities of the Financial Information eXchange (FIX) protocol. During development, testing, or production support, it's often necessary to inspect past FIX messages. QuickFIX logs all incoming and outgoing messages, providing a valuable historical record. This article will guide you through the process of loading and replaying these messages from your QuickFIX logs, which is crucial for debugging issues, re-creating scenarios, or analyzing message flows.

Understanding QuickFIX Log Formats

QuickFIX typically stores messages in a human-readable format within its session logs. These logs are usually found in a directory specified in your QuickFIX configuration (e.g., FileLogPath). Each session has its own log files, often named with a timestamp or sequence number. The messages themselves are stored as raw FIX strings, sometimes prefixed with direction indicators (e.g., 8=FIX.4.2).

It's important to distinguish between the raw message log and the event log. The raw message log contains the actual FIX messages exchanged, while the event log records internal QuickFIX events like session establishment, disconnection, and errors. For replaying messages, we are primarily interested in the raw message logs.

QuickFIX Logging and Message Flow

Extracting FIX Messages from Log Files

The first step in loading messages is to extract them from the log files. QuickFIX logs can contain various entries, so you'll need to filter for actual FIX messages. A common pattern for FIX messages is the presence of the 8=FIX.X.X tag at the beginning of a line. You can use simple scripting or command-line tools to achieve this extraction.

grep -oE '8=FIX\.[0-9]\.[0-9].*' quickfix_session.log > extracted_fix_messages.txt

Get-Content quickfix_session.log | Select-String -Pattern '8=FIX\.[0-9]\.[0-9].*' | Out-File extracted_fix_messages.txt

Parsing and Replaying Messages Programmatically

Once you have a collection of raw FIX messages, you can parse them using the QuickFIX Message class. This allows you to inspect individual fields, modify messages, or even replay them through a QuickFIX session. Replaying messages is particularly useful for testing how your application responds to specific message sequences or error conditions.

C++

#include <quickfix/Message.h>
#include <quickfix/FixFields.h>
#include <quickfix/FixValues.h>
#include <iostream>
#include <fstream>
#include <string>

int main() {
    std::ifstream infile("extracted_fix_messages.txt");
    std::string line;

    while (std::getline(infile, line)) {
        try {
            FIX::Message message;
            message.setString(line);

            // Example: Print message type and sender/target comp IDs
            FIX::MsgType msgType;
            message.getHeader().getField(msgType);
            FIX::SenderCompID senderCompID;
            message.getHeader().getField(senderCompID);
            FIX::TargetCompID targetCompID;
            message.getHeader().getField(targetCompID);

            std::cout << "Parsed Message: MsgType=" << msgType.getValue() 
                      << ", SenderCompID=" << senderCompID.getValue()
                      << ", TargetCompID=" << targetCompID.getValue() << std::endl;

            // To replay, you would typically get a session and send the message:
            // FIX::Session::sendToTarget(message);

        } catch (const FIX::Exception& e) {
            std::cerr << "Error parsing message: " << e.what() << " - " << line << std::endl;
        }
    }
    return 0;
}

Java

import quickfix.Message;
import quickfix.FieldNotFound;
import quickfix.StringField;
import quickfix.fix42.NewOrderSingle;
import quickfix.field.MsgType;
import quickfix.field.SenderCompID;
import quickfix.field.TargetCompID;

import java.io.BufferedReader;
import java.io.FileReader;
import java.io.IOException;

public class LogMessageParser {

    public static void main(String[] args) {
        try (BufferedReader reader = new BufferedReader(new FileReader("extracted_fix_messages.txt"))) {
            String line;
            while ((line = reader.readLine()) != null) {
                try {
                    Message message = new Message(line);

                    // Example: Print message type and sender/target comp IDs
                    MsgType msgType = new MsgType();
                    message.getHeader().getField(msgType);
                    SenderCompID senderCompID = new SenderCompID();
                    message.getHeader().getField(senderCompID);
                    TargetCompID targetCompID = new TargetCompID();
                    message.getHeader().getField(targetCompID);

                    System.out.println("Parsed Message: MsgType=" + msgType.getValue() 
                                       + ", SenderCompID=" + senderCompID.getValue()
                                       + ", TargetCompID=" + targetCompID.getValue());

                    // To replay, you would typically get a session and send the message:
                    // Session.sendToTarget(message);

                } catch (FieldNotFound e) {
                    System.err.println("Field not found in message: " + e.getMessage() + " - " + line);
                } catch (Exception e) {
                    System.err.println("Error parsing message: " + e.getMessage() + " - " + line);
                }
            }
        } catch (IOException e) {
            e.printStackTrace();
        }
    }
}

Python

import quickfix as fix

def parse_and_replay_messages(log_file_path):
    with open(log_file_path, 'r') as f:
        for line in f:
            try:
                message = fix.Message(line.strip())

                # Example: Print message type and sender/target comp IDs
                header = message.getHeader()
                msg_type = header.getField(fix.MsgType())
                sender_comp_id = header.getField(fix.SenderCompID())
                target_comp_id = header.getField(fix.TargetCompID())

                print(f"Parsed Message: MsgType={msg_type}, SenderCompID={sender_comp_id}, TargetCompID={target_comp_id}")

                # To replay, you would typically get a session and send the message:
                # fix.Session.sendToTarget(message)

            except fix.FixError as e:
                print(f"Error parsing message: {e} - {line.strip()}")

if __name__ == "__main__":
    parse_and_replay_messages("extracted_fix_messages.txt")

Advanced Considerations for Message Replay

Replaying messages isn't always as simple as sending them one by one. Consider the following:

• Sequence Numbers: FIX messages rely heavily on sequence numbers. When replaying, you might need to adjust sequence numbers to match the current state of the session or configure your QuickFIX initiator/acceptor to ignore sequence number checks for testing purposes.
• Timestamps: Timestamps (e.g., SendingTime tag 52) are critical. If you replay old messages, their timestamps will be in the past. You might need to update these timestamps to the current time before sending, especially if your counterparty or application has strict timestamp validation.
• Session State: A QuickFIX session maintains state. Replaying messages out of context (e.g., sending an ExecutionReport without a preceding NewOrderSingle) can lead to errors or unexpected behavior. For complex scenarios, consider building a mock counterparty or a dedicated replay tool that understands session state.
• Message Modification: You might want to modify certain fields (e.g., ClOrdID, OrderID, Price, Quantity) in the replayed messages to test different scenarios without altering the original log content.

1. Locate QuickFIX Log Files

Identify the FileLogPath in your QuickFIX configuration. This directory contains the session-specific log files (e.g., FIX.4.2-SENDER-TARGET.messages.log).

2. Extract Raw FIX Messages

Use grep (Linux/macOS) or Select-String (Windows PowerShell) to filter for lines starting with 8=FIX and redirect them to a new file.

3. Implement Message Parsing Logic

Write a program in C++, Java, or Python using the QuickFIX library to read the extracted messages line by line.

4. Parse and Inspect Messages

Within your program, use quickfix.Message to parse each raw FIX string. Access header and body fields to inspect message content.

5. Consider Message Modification (Optional)

If needed, modify fields like SendingTime, ClOrdID, or OrderID before replaying to adapt to current test conditions.

6. Replay Messages (Test Environment)

Obtain a reference to an active QuickFIX session (e.g., Session.sendToTarget(message)) and send the parsed messages. Ensure this is done in a controlled test environment.