Implementing Auto focus for home made microscope

One main difficulty in using home made microscope is that image is getting out of focus so easily, especiallly in high magnification. This time I implemented autofocus function to solve this problem.

 

1. Example of system stabilization effect on the focus

Because of many reasons, image can be defocused. One of main reason is the change in the temperature which results in thermal expansion. When temperature changes, components of microscope are expanding or shrinking and it results in the change of distance between the sample and objective lens. As a result, the image gets defocused. Below images are the example of image is getting defocused over time. Experiment started at 22:29. every 10 seconds, sample is rotated by 60 degrees and image is grabbed. After a few minutes, image gets defocused, because the rotation motor gets heated up. 

Images in the beginning of the experiment. Image gets defocused in a few minutes because of temperature change in the rotation motor.

 

During the experiment, I stopped the script and adjusted the z stage quickly to focus the image. script was executed again after the adjust. As you can see on the below, focus doesn't change like the beginning of the experiment. it means that focus is not fluctuating when system gets stabilied. 

 

The example above shows that the needs for the autofocus function; focus can change if system gets disturbed. and my system can be easily disturbed by any change. the temperature change between day to night/human activity/window opening/ etc..

 

2. Implementing Autofocus - HW side

on HW side, I need to actuate z motor which is assembled inthe microscope. I found that the controller of microscope include driver for the z stage, so I could utilize it.

 

 

To operate the driver, I need i) to put +24V, +5V to driver (CN1)  and ii)put CW and CCW pulses to driver to actuate motor. (CN2, pin1, 2) For the +24V, +5V, I used the power supply of PC. +12V, -12V are used to created +24V.for CW and CCW pulse, I used Arduino PWM signal. PWM of arduino is directly connected to CW and CCW pin, then duration of signal is changed to control the amount of movement in z stage. After configuring the electronics, I could move z stagfe by apply voltage to CW or CCW.

 

As like the video, I could move up or down the motor. (although it doesn't look so good now..)

https://youtube.com/shorts/1r2RigEC1WI?feature=share

 

 

4. Implementing Autofocus - SW side

I used python as main control part. Image is grabbed using python and motors (rotation and z) are controlled from python by serial communication to arduino. 

python code is simple. It grabs the image and sample rotate for every image. and AF was executed for every 6th image. When python sent message like "r01" to arduino, arduino rotate by 60 deg. Auto focus is implemented as separate function. 

 

# loop for experiment
while True :
    timeNow = datetime.datetime.now()
    timeDiff = startTime - timeNow
    timeDiffSec = timeDiff.seconds

    # grab 6 images while rotating
    if timeDiffSec % timeGap == 0 :
        print(f'time diff : {timeDiffSec}, status : {status}')
        
        #do AF for every 6th images
        if status % 6 == 0:
            print(f"do AF")
            AF.doAF2(py_serial, rangeAll= 40, step = 4)
            time.sleep(3)
            print(py_serial.read_all())
            py_serial.close()
            py_serial.open()
            time.sleep(2)
        cap = cv2.VideoCapture(0)

        # rotate sample
        py_serial.write(f"r01".encode())
        print(py_serial.read_all())
        time.sleep(2)
        print(py_serial.read_all())
        capture_image(cap)
        status = status+1

        # py_serial.close()
        cap.release()
    else :
        time.sleep(1)

 

To do Autofocus, z stage sweeps the range, and sharpness is calculated during the sweep. in the end, z stage is positioned to the location where sharpness was highest. sharpness is calculated by simple Laplacian function. 

One difficulty I have was that serial connumication between arduino and python was unstable. when I run the code with short delay bewteen each function, the connection get lost some how. I mitigated it by adding several seconds between each operation. It was quite strage because when I did same thing on Arduino IDE, connect never get lost. I didn't dig into this because it's not in my interest.

Down side of this implementation is that it's very slow. 

def doAF2(py_serial, rangeAll = 20, step = 2):

    cap = cap1()

    start = int(rangeAll/2)
    num = int(rangeAll/step)
    delayForCommunication = 2

    # py_serial.write(f'z2{parseIntToMessage(start)}'.encode())
    time.sleep(1)
    serialWrite(py_serial, f'z2{parseIntToMessage(start)}')
    time.sleep(delayForCommunication)
    # move z and calculate sharpness
    sharpnessList = []
    for i in range(num):
        # py_serial.write(f'z1{parseIntToMessage(step)}'.encode())
        serialWrite(py_serial,  f'z1{parseIntToMessage(step)}')
        print(py_serial.in_waiting, py_serial.read_all())
        ret, frame = cap.read()
        L = frame[:, :, 0]
        LP = cv2.Laplacian(L, cv2.CV_16S, ksize=3)
        s = np.sum(np.abs((LP))) / np.sum(L)
        sharpnessList.append(s)

        time.sleep(delayForCommunication)

    maximumPosition = np.argmax(sharpnessList)

    # move to the final position
    # py_serial.write(f'z2{step*(num-maximumPosition)}'.encode())
    serialWrite(py_serial,  f'z2{step*(num-maximumPosition)}')
    time.sleep(delayForCommunication)
    py_serial.reset_output_buffer()
    py_serial.reset_input_buffer()
    py_serial.flush()

    ret, frame = cap.read()
    cap.release()
    # py_serial.close()
    print(f"do AF2 done, sharpness : {sharpnessList[maximumPosition]}")
    return maximumPosition, sharpnessList, frame

 

On Arduino, I wrote the code like below. 

const int ENA = 11;
const int IN1 = 12;
const int IN2 = 13;

const int zCW = 5;
const int zCCW = 6;

int rStatus;
char message[5]; // To store the incoming message

int rotDelay = 1500;

void setup() {
  Serial.begin(9600);
  pinMode(ENA, OUTPUT);
  pinMode(IN1, OUTPUT);
  pinMode(IN2, OUTPUT);
  rStatus = 1;
  analogWrite(ENA, 255);
}

void loop() {
  // Check if there is data available
  if (Serial.available() > 0) {
    Serial.readBytes(message, 5); // Read the message into the array
    //message[4] = '\0'; // Null-terminate the string

    processMessage(message);
  }
}

// Function to process the message and send signal to the motor driver
void processMessage(char* msg) {
  char motorType = msg[0]; // Extract the first character (z or r)
  int movementValue = atoi(&msg[1]); // Convert the next three characters to an integer

  switch (motorType) {
    case 'z':
      driveMotorZ(movementValue);
      break;
    case 'r':
      driveMotorR(movementValue);
      break;
    default:
      Serial.println("Unknown motor type.");
      break;
  }
}

// Example functions to drive different motors
void driveMotorZ(int value) {
  // Add code to control motor Z with 'value'

  char direction = String(value)[0];
  int duration = atoi(&String(value)[1]);
  Serial.println("z operation");
  Serial.println(direction);

  switch (direction){
    case '1':
      analogWrite(zCW, 50);
      delay(duration*2);
      analogWrite(zCW, 0);
      Serial.println("z up");
      break;
    case '2':
      analogWrite(zCCW, 50);
      delay(duration*2);
      analogWrite(zCCW, 0);
      Serial.println("z down");
      break;
  }

  analogWrite(zCCW, 0);
  analogWrite(zCW, 0);

  Serial.print("Driving motor Z with value: ");
  Serial.println(value);
  // For example: analogWrite(Z_MOTOR_PIN, value);
}

void driveMotorR(int value) {
  // Add code to control motor R with 'value'
  Serial.print("Driving motor R with value: ");
  Serial.println(value);
  // For example: analogWrite(R_MOTOR_PIN, value);

  while (value > 0){

    switch (rStatus){
      case 1:
        digitalWrite(IN1, HIGH);
        digitalWrite(IN2, LOW);
        rStatus = 2;
        delay(rotDelay);
        //Serial.println(rStatus);
        break;
      case 2:
        digitalWrite(IN1, LOW);
        digitalWrite(IN2, HIGH);
        rStatus = 1;
        delay(rotDelay);
        //Serial.println(rStatus);
        break;
      default:
        break;
    }
      digitalWrite(IN1, LOW);
      digitalWrite(IN2, LOW);
    value = value-1;
    Serial.println(value);
  }

  

}

 

 

5. Measurement with Autofocus

I run the experiment with AF, below images are the results. Even In the beginning of the experiment, image doesn't get defocused. it seems like it's working well.