Technical help please

fluidfcs1 · April 21, 2023, 1:24pm

Hi folks, I recently got my hands on an MED100KX8 100 kHz device (Thanks Scott). I’ve invested so far an immense amount of time in trying to get it working, and could use some help.

I think it’s important to first highlight that I am using a Macbook Pro with an M2 chip. The M2 chips use an ARM64 architecture, which generally results in lots of compatibility issues with older software as they only have x86 or x64 drivers (sometimes even i386, but rarely arm64). Natively I am running macOS 13.3.1, but using Parallels have created virtual machines that run Windows 11 and Ubuntu Linux 22.04. I’d ideally like to get the RNG working in macOS, but at this point would settle for getting it working anywhere.

Summary of where I’m at by operating system:

Windows 11: I can successfully run QNG360_SETUP.exe, but get an error that QWQNG.DLL failed to load. The QNG Meter program opens but am not able to get it to find the device. When I try to run either QNG File Maker or QNG Report a terminal window pops up and immediately closes. I can see the MED100KX8100 device in Device Manager with a driver error. When I check the driver options there are various architecture options but none for ARM64. I can successfully load generic SB Serial Converter drivers, but that doesn’t really help me run any of the programs (QNGmeter still finds no devices). Interestingly, I tried this also on an older Macbook that has an Intel (x86) chip. I used Bootcamp to install Windows 10. I am seeing all the same behaviors (only able to load generic serial drivers).

macOS: I tried to manually build libqwqng in macOS. After modifying the makefile and manually copying ftdi.h I was able to get cmake to complete, but then the subsequent “make” command fails due to missing libftdi1.so and libusb-1.0.so in /usr/local/lib/. I was never able to resolve this, even after using brew to install packages: libftdi, libftdi0, libfido2, libomp, llvm, libusp, and also manually building and installing the libftdi and libusb packages in macOS. I noticed that the two missing files (libftdi1.so and libusb-1.0.so) were present in my Ubuntu instance and so copied them over to macOS but then received errors for unsupported file formats and lots of missing references. Stuck there.

Ubuntu: The linux software for libqwqng contains a script to build/install packages for Ubuntu, which is why I went with Ubuntu. After working through a number of errors I got the script to complete successfully and was able to manually build libqwqng (which I never succeeded in doing in macOS). However, what I’m left with are C++ files (not compiled files as the documentation suggests), and so far have not been able to get the C++ files to compile using G++. I’ve worked through at least a dozen different errors but in the end am stuck on this error: here.

Interestingly, if I copy the entire directly from Ubuntu over to macOS and try to compile there with G++ (using the qwqng build done by Ubuntu) then I get very similar errors, but fewer. That looks like this.

Starting now to get a little creative, and in response to one of Scott’s suggestions, I head back to macOS and try a different approach. Perhaps I can avoid using libqwqng altogether and directly interact with the device. I have two other RNG devices that work by simply using Python code to listen for input on the serial port. I am able to connect to the MED100KX8100 device like this also, but it produces no output. It seems that, unlike my other devices, I need to send some command to have the device transmit data. I expect I need to use some FTDI library for this. I installed the following packages: ftdi-vcp-driver, pyftdi, ftd2xx, pylibftdi. I’m able to use each of this in different ways to successfully connect to the device and get device info, serial, product ID, etc. So that’s a good sign.

Interestingly, one of them in particular seems to cause data to start flowing immediately after connection. This Python code…

import pyftdi.serialext

# Open a serial port on the second FTDI device interface (IF/2) @ 3Mbaud
port = pyftdi.serialext.serial_for_url('ftdi://ftdi:232:QWR4E010/1', baudrate=3000000)

# Receive bytes
data = port.read(1024)
print(f'{data}')

…produces output that looks like this. That looks like bytecode, and should be bytecode, but whenever I try to convert it to ASCII or string, or anything readable it fails for syntax errors.

Looking through the various sources files I found qwqng.hpp, which seems to have raw instructions for sending to the FTDI device. For example line 48 is:

#define FTDIDEVICE_START_COMMAND_           0x96U

I tried sending some of these commands via the .write function but I wasn’t seeing anything happen different.

So, my questions…

Do I need to send these commands, and if so using what syntax?
Is the bytecode I’m receiving really bytecode, and if so can I use this somehow?
Any other advice for getting qwqng working on any of my three operating systems rather than trying to hack it like this?

I think I’m basically stuck at this point and would really appreciate any help. Thanks in advance.

WanderingIshiki · April 22, 2023, 3:36am

I created a driver called Meter Feeder which also works with Scott’s hardware. You could try that:

fluidfcs1 · April 22, 2023, 2:16pm

@WanderingIshiki Thanks for this. Scott actually just shared the same. Trying to work through this but I admit I am out of my depth. What I’m hoping to be able to do is use Python to read numbers generated by the device that I can use in Python programs. The device is connected via USB->USB-C converter.

I am able to load, connect, view device info with all of the following libraries:

But I can’t seem to figure out how to pull actual numbers. I can get data with the read function from ftd2xx or ftdi_fn.ftdi_read_data from pylibftdi but it’s just the ASCII(?) character or bytecode(?) repeated over and over (e.g. ÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿ or /xfe/xfe/xfe/xfe) for the length of characters I pass to the function.

It feels like I am very close here but not sure where to go. Is there perhaps some command I need to send to the device to have it start producing numbers? In the source you shared it seems like ftd2xx.h is what I need to be looking at, but I’m really not sure how to pass these instructions to the device. For example, when I see something in the file like the below, I have no idea how to actually do something with this. How would I the device that I want this type of flow control (not that I actually do)?

#define FT_FLOW_RTS_CTS		0x0100

I’m hoping that puling numbers from this device can be as simple as a few commands I can hardcode to pass (e.g. open connection, set baudrate, set data format), then can just pull data at my leisure with some command. I really don’t know what I’m doing here. Any chance you could give me more specific guidance ideally in the form of a few Python commands?

When I run this:

import sys, ftd2xx as ftd
d = ftd.open(0)  
print(d.getDeviceInfo())

It returns: {‘type’: 5, ‘id’: 67330049, ‘description’: b’MED100KX8 100 kHz’, ‘serial’: b’QWR4E010’}

If I then run:

data = d.read(100)
print(f'{data}')

It usually returns nothing but I notice that sometimes when I’m trying to interact with the device using other libraries it causes data to be returned by d.read(100)… but it’s just things like /xfe/xfe/xfe… of /xff/xff/xff. A couple days ago, as I posted, I was getting a large variety of different characters ( here), but I’m not able get that type of output anymore and I don’t know why.

Appreciate your help.

ScottWilber · April 22, 2023, 4:00pm

Hi Dan,
I will try to recall the handshaking that happens when running the interface program, but it’s been a long time and I didn’t write any of the code myself.

When the interface program is run it tries to identify one of our connected device by looking at the serial number. Only the first 4 digits are used: that is always QWRx, where x is a number; 4 includes all the MED100Kxx devices, 6 is for 4Mbps PQ devices, 7 is for 128Mbps PQ devices, 8 is for 128Mbps Crypto. After a connected device is identified, a command is sent to the device to turn it on, something like binary 2. Then the device should be continuously producing output to be read by appropriate commands in the interface software.

When more than one device is connected, the interface will connect in numerical sequence using the last digits of the serial number. I don’t recall if that is from lowest to highest or vice versa.

I hope this is some help.

fluidfcs1 · April 22, 2023, 4:24pm

I can connect and send read / write commands. If I do that with the ftd2xx library I get bytecode, and always one of three outcomes

No output (this is what I get usually, but as mentioned after interacting with other libraries I can start seeing output here, they must initialize the device in a way that this library does not)
Repeating \xff\xff\xff\xff…
Repeating \xfe\xfe\xfe\xfe…

These are bytecode representations of ÿ and þ. The number I pass in my read function is the number of bytecode characters I get (e.g. read(3) gives me b’\xff\xff\xff’).

If I use the pylibftdi library instead and but the text in text mode then I get the ASCII characters and not the bytecode (ÿ and þ). I can’t say why I get one of the other, but I can only return one exclusively for long periods of time and then sometimes it changes.

fluidfcs1 · April 22, 2023, 5:48pm

Update, I’ve combined the use of both the ftd2xx and pylibftdi libraries, since I seem to be able to initialize the device with pylibftdi but @WanderingIshiki 's code uses ftd2xx.

By sending either d.write(b"0x00") or d.write(b"0x01"), and then resetting the USB, I can switch the data I get back from being entirely /xfe (þ) or entirely /xff (ÿ). Still stuck, but thought this might be useful info to anyone able to give me advice.

import sys
import ftd2xx as ftd
from pylibftdi import Device

with Device(mode='t') as d1:  # opens device with pylibftdi
	d1.baudrate = 115200
	OP = 0                              # Bit 0 will be an output
	d1.ftdi_fn.ftdi_set_bitmode(OP, 1)  # Return 0 if bitbang mode
	d1.ftdi_fn.ftdi_usb_reset()
	output=d1.read(100)
	print(f'pylibftdi output: {output}')

d = ftd.open(0)    # Open first FTDI device with ftd2xx
print(d.getDeviceInfo())

d.write(b"0x01") # makes /xff or ÿ (after usb_reset)
#d.write(b"0x00") # makes /xfe or þ (after usb_reset)

data = d.read(100)
print(f'ftd2xx output: {data}')

output… (with 0x01)

pylibftdi output: ÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿ

{'type': 5, 'id': 67330049, 'description': b'MED100KX8 100 kHz', 'serial': b'QWR4E010'}

ftd2xx output: b'\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff'

ScottWilber · April 22, 2023, 8:45pm

Here is my best guess at Python code for opening/reading the MED100Kxx device. Note that this code is set to read 8 bytes at a time – a purely arbitrary number, which can be changed:

import sys
from pyftdi.ftdi import Ftdi
from pyftdi.usbtools import UsbTools

FTDI_DEVICE_LATENCY_MS = 2
FTDI_DEVICE_PACKET_USB_SIZE = 8
FTDI_DEVICE_TX_TIMEOUT = 5000

def device_find():
    devices = Ftdi.find_all([(0x0403, 0x6014)], nocache=True)
    for device in devices:
        serial_number = device[2]
        if serial_number.startswith("QWR4"):
            return serial_number
    return ""

def device_startup(serial_number):
    if not serial_number:
        print("%%%% device not found!")
        return None

    print("deviceId:", serial_number, "\n")
    ftdi = Ftdi()
    ftdi.open_from_url(f"ftdi://{serial_number}")

    ftdi.set_latency_timer(FTDI_DEVICE_LATENCY_MS)
    ftdi.write_data_set_chunksize(FTDI_DEVICE_PACKET_USB_SIZE)
    ftdi.read_data_set_chunksize(FTDI_DEVICE_PACKET_USB_SIZE)
    ftdi.set_timeouts(FTDI_DEVICE_TX_TIMEOUT, FTDI_DEVICE_TX_TIMEOUT)

    return ftdi

def device_shutdown(ftdi):
    ftdi.close()

def main():
    serial_number = device_find()
    ftdi = device_startup(serial_number)

    if not ftdi:
        input("Press any key to exit.")
        sys.exit(-1)

    init_comm = b'\x96'
    ftdi.purge_buffers()
    bytes_txd = ftdi.write_data(init_comm)

    if bytes_txd != len(init_comm):
        print("%%%% Write Failed!")
        sys.exit(-1)

    bytes_to_rx = 8
    dx_data = ftdi.read_data(bytes_to_rx)

    if len(dx_data) != bytes_to_rx:
        print("%%%% Read Failed!")
        sys.exit(-1)

    for byte in dx_data:
        print(f"{byte:02X}", end=" ")

    device_shutdown(ftdi)

if __name__ == "__main__":
    main()

Before running the code, make sure to install the PyFtdi package using pip:

pip install pyftdi

This code should work similar to the original C++ code.

NOTES:
The Python code should work on both macOS and a Windows 11 virtual machine using Parallels, with a few caveats:

On macOS, you might encounter issues with the FTDI VCP (Virtual COM Port) drivers. macOS has built-in VCP drivers that might claim the FTDI device and prevent the PyFtdi library from accessing the device. To resolve this issue, you can unload the macOS VCP driver with the following command:

css

sudo kextunload -b com.apple.driver.AppleUSBFTDI

In a Windows 11 virtual machine using Parallels, the FTDI device might not be automatically passed through to the virtual machine. You need to ensure that the FTDI device is connected to the virtual machine. In Parallels, you can configure the USB settings to make sure the FTDI device is connected to the virtual machine when plugged in.
On both macOS and Windows, you need to make sure that you have the appropriate permissions to access the USB devices. On macOS, you might need to run the script with sudo if you encounter permission issues. On Windows, running the script with administrator privileges should resolve any permission-related issues.
Before running the script on macOS, install the necessary dependencies via pip. For Windows, you might need to install the appropriate drivers for the FTDI device. The official FTDI website provides the necessary drivers for Windows: VCP Drivers - FTDI
Ensure that the PyFtdi package is installed on both macOS and the Windows 11 virtual machine using:

pip install pyftdi

fluidfcs1 · April 23, 2023, 3:29pm

Thank you Scott for taking the time to write this. I had a few issues when trying to execute but have been using ChatGPT to debug and modify. I ended up hardcoding a few values to work around errors. It’s now producing a number. The number seems too large though. For example: 315549251183339525

Here’s the code:

import sys
from pyftdi.ftdi import Ftdi

FTDI_DEVICE_LATENCY_MS = 2
FTDI_DEVICE_PACKET_USB_SIZE = 8
FTDI_DEVICE_TX_TIMEOUT = 5000

def device_startup(serial_number):
    if not serial_number:
        print("%%%% device not found!")
        return None

    print("deviceId:", serial_number, "\n")
    ftdi = Ftdi()
    #ftdi.open_from_url(f"ftdi://{serial_number}/1")  # Added device port
    ftdi.open_from_url(f"ftdi://ftdi:232:{serial_number}/1") # hard code after checking Ftdi.show_devices()
    ftdi.set_latency_timer(FTDI_DEVICE_LATENCY_MS)
    ftdi.write_data_get_chunksize = lambda x: FTDI_DEVICE_PACKET_USB_SIZE
    ftdi.read_data_get_chunksize = lambda x: FTDI_DEVICE_PACKET_USB_SIZE

    return ftdi

def device_shutdown(ftdi):
    ftdi.close()

def main():
    serial_number = "QWR4E010"  # Replace with your serial number
    ftdi = device_startup(serial_number)

    if not ftdi:
        input("Press any key to exit.")
        sys.exit(-1)

    init_comm = b'\x96'
    ftdi.purge_buffers()
    bytes_txd = ftdi.write_data(init_comm)

    if bytes_txd != len(init_comm):
        print("%%%% Write Failed!")
        sys.exit(-1)

    bytes_to_rx = 8
    dx_data = ftdi.read_data(bytes_to_rx)

    if len(dx_data) != bytes_to_rx:
        print("%%%% Read Failed!")
        sys.exit(-1)

    received_int = int.from_bytes(dx_data, "big")
    print(received_int)

    device_shutdown(ftdi)

if __name__ == "__main__":
    main()

I had trouble with the chunksize functions not working and you can see how they are modified. Maybe this causes some problem.

ScottWilber · April 23, 2023, 4:01pm

315549251183339525 is a 59-bit integer in binary – possibly all eight 8-bit bytes called by the function combined into one number with 5 leading 0s. This number in binary is:
10001100001000011100101111101000001111011000110100000000101. This one example doesn’t appear quite random – some longish runs – but it’s not clearly non-random either. Try the following:

To return the bytes individually as integers, you can return the dx_data bytearray or convert it to a list of integers. This example reads data from the device and returns the bytes as a list of integers:

def read_device_data(ftdi, bytes_to_rx):
dx_data = ftdi.read_data(bytes_to_rx)

if len(dx_data) != bytes_to_rx:
print(“%%%% Read Failed!”)
return None

return list(dx_data)

fluidfcs1 · April 23, 2023, 4:05pm

examples of the new output with your suggested changes:
16679233395028652763
13399151302405245649
12591792871996962893
13428831024224813184

import sys
from pyftdi.ftdi import Ftdi

FTDI_DEVICE_LATENCY_MS = 2
FTDI_DEVICE_PACKET_USB_SIZE = 8
FTDI_DEVICE_TX_TIMEOUT = 5000

def device_startup(serial_number):
    if not serial_number:
        print("%%%% device not found!")
        return None

    print("deviceId:", serial_number, "\n")
    ftdi = Ftdi()
    #ftdi.open_from_url(f"ftdi://{serial_number}/1")  # Added device port
    ftdi.open_from_url(f"ftdi://ftdi:232:{serial_number}/1") # hard code after checking Ftdi.show_devices()
    ftdi.set_latency_timer(FTDI_DEVICE_LATENCY_MS)
    ftdi.write_data_get_chunksize = lambda x: FTDI_DEVICE_PACKET_USB_SIZE
    ftdi.read_data_get_chunksize = lambda x: FTDI_DEVICE_PACKET_USB_SIZE

    return ftdi

def device_shutdown(ftdi):
    ftdi.close()

def read_device_data(ftdi, bytes_to_rx):
    dx_data = ftdi.read_data(bytes_to_rx)

    if len(dx_data) != bytes_to_rx:
        print("%%%% Read Failed!")
        return None

    return list(dx_data)

def main():
    serial_number = "QWR4E010"  # Replace with your serial number
    ftdi = device_startup(serial_number)

    if not ftdi:
        input("Press any key to exit.")
        sys.exit(-1)

    init_comm = b'\x96'
    ftdi.purge_buffers()
    bytes_txd = ftdi.write_data(init_comm)

    if bytes_txd != len(init_comm):
        print("%%%% Write Failed!")
        sys.exit(-1)

    bytes_to_rx = 8
    dx_data = read_device_data(ftdi, bytes_to_rx)

    if not dx_data:
        sys.exit(-1)

    received_int = 0
    for byte in dx_data:
        received_int = (received_int << 8) + byte

    print(received_int)

    device_shutdown(ftdi)

if __name__ == "__main__":
    main()

ScottWilber · April 23, 2023, 4:19pm

The four numbers are all 64-bits in binary form.
The first number in binary is:
1110011101111000100010100110011100000100100000111111001011011011

These do appear to be the eight 8-bit random bytes being returned, but displayed as a single integer. Leading 0s are valid random binary bits, so they must always be retained. I don’t know if that is an issue in Python.

Note, in time sequence, bits are provided in 8-bit bytes, LSB first to MSB last.

Seems to be working now!

fluidfcs1 · April 23, 2023, 4:36pm

Amazing, thank you so much for your help!