ESP8266 resets when serial monitor is opened via a FT232 USB UART Board

[Vitotje]

DTR and RTS pins are set to GND as long as the serial monitor is opened via a FT232 USB UART Board regardless the checkbox setting in the 'add serial port' dialog window.
Programming the EPS8266 via this FT232 USB UART Board works perfectly.

An official NodeMCU EPS8266 board with USB-UART included, does not show this problem.
The serial monitor of the Arduino IDE does not have this problem.

Used setup is the installation of Sloeber V4.4.1_win64.2022-09-05_06-48-31.zip on a Windows 10 and Windows 11 PC.

This problem is very similar as mentioned in reported issue 'ESP8266 resets when serial monitor is opened #394'

[jantje]

I have read your post a couple of times but I can't make sense of it.
What do you mean by "opened"
What is the problem?
If it is very similar to #394 what is the difference?

[Vitotje]

Sorry for my unclear description.
Hopefully this explains better:

With "opened" I mean as long as I do not open the serial monitor functionality in Sloeber, the DTR and RTS pins of the FT232 USB UART Board connected to the ESP8266 chip are not activated (high level) and so, the ESP8266 is normal operating. Once when I open the serial monitor to see the output of the EPS8266, these pins get activated (low level) and the ESP8266 is kept in the reset mode.

The problem is that I can't use the serial monitor of Sloeber because my hardware has a FTDI chip as USB-serial interface and Sloeber is activating the ESP8266 reset when I open the serial monitor.

The difference with #394 is that once when I have an ESP8266 NodeMCU panel connected, Sloeber controls the DTR and RTS lines correctly and what I understand, this is not in problem description #394.

It looks like Sloeber does not correctly control the serial port for different types of USB-serial hardware.
For me, remarkable because Sloeber is able to download the firmware correctly via the FTDI chip and for that he also needs to control the DTR and RTS lines!
Also remarkable is that the Arduino IDE can handle the different hardware so I expect some missing lines of code in Sloebers serial setup (this also happens to me sometimes).

At the moment, as work around, I use Sloeber to design and check my code and Arduino IDE to compile, upload and see the serial monitor.

Thanks for your great Sloeber design, a huge improvement compared to the Arduino IDE.

[jantje]

I'm still not fully understanding but I think this is about a physical board connecting the pc with a board (named FT232 USB UART Board) not having a serial interface.
This board seems to have problems with the serial settings Sloeber uses.
Given I do not own a "FT232 USB UART Board" I see 3 ways to fix this problem.

1. I get one of these boards (don't like that as I already have to much stuff lying around)
2. I do the changes and send them to you; you try (and repeat untill it works) (I don't like this as I don't know what to change)
3. You do the changes and send them to me; I work from there.

3 is the best solution to me as then at least I know what to change.
I think changes should be applied at port creation (https://github.com/Sloeber/arduino-eclipse-plugin/blob/master/io.sloeber.core/src/io/sloeber/core/api/Serial.java#L89) or port connection (https://github.com/Sloeber/arduino-eclipse-plugin/blob/master/io.sloeber.core/src/io/sloeber/core/api/Serial.java#L178)
It is pretty simple straight forward code.
If you could debug you could set a breakpoint at setDTR https://github.com/Sloeber/arduino-eclipse-plugin/blob/master/io.sloeber.core/src/io/sloeber/core/api/Serial.java#L303 and setRTS https://github.com/Sloeber/arduino-eclipse-plugin/blob/master/io.sloeber.core/src/io/sloeber/core/api/Serial.java#L313
This could tell us wether we are looking for code removal or code addition.

Do you see this as feasible?

[Vitotje]

I found the root cause of the problem. Sloeber supports only ESP hardware as implemented in the commercial NodeMCU boards.
I have tested a proposal how to improve Sloeber so it supports NodeMCU designs as well as ESP hardware that have directly connected the serial RTS and DTR pints to the EPS8266.
You can find my recommendation at the end of this (long) story.

More in depth.
I went for your proposal 3 and put your io.sloeber.core.api.Serial in my local test bench for investigation.
Immediately after calling serial.connect() I saw different hardware behavior between the commercial NodeMCU board and my ESP design: NodeMCU boots but my design stuck in reset mode.
Measuring the RTS and DTR signal levels coming from the two different brands of USB-serial interfaces I used, did not show any differences! This triggers me to start searching for the electrical diagram of the NodeMCU to investigate the electronics they put on that board.

In attached figure 'nodemcu-circuit-diagram.jpg' you see that the DRT and RTS pins of USB-serial interface U2 via two transistors VT1 and VT2 are connected to the EPS-RST and EPS-GPIO0. This circuitry is more or less a XOR configuration, see the small table in that diagram just above this circuitry.
One very important component that belongs to this circuitry is the 470pF capacitor C2 connected to the RST line. Without this C2 it is not possible to enter the serial download mode of the ESP8266! This will be explained later.

Without serious knowledge of electronics it is hard to understand what's going on so I made a timing diagram 'ESP_reset.jpg' where you can see the relation between the logic levels of RTS and DTR coming out of the USB interface U2 and the levels of RST and GPIO0 going to the EPS8266.
Extra information you need to know is that the ESP8266 clocks (read 'implements') the desired mode at the rising edge of the ESP_RST. As example, supose ESP_GP0 is LOW and ESP_RST goes from LOW to HIGH, only then the EPS8266 goes into the serial download mode.

Explanation of the timing diagram.
Basically said: signal RTS is only passed through to ESP_RST when DTR is HIGH and signal DTR is only passed through to ESP_GP0 when RTS is HIGH. In all other situations signals EPS_RST and EPS_GP0 are HIGH.
But in the timing diagram we see an exception called 'delay'. This is an important delay in ESP_RST when RST rises from LOW to HIGH. The cause of this delay is capacitor C2.
This delay makes it possible to enter the serial download mode as can be seen in situation 1 and 2. In these situations ESP_RST goes HIGH while ESP_GP0 is LOW and so entering the download mode.

What's going wrong with Sloeber connected to my design.
That is, when opening the serial monitor, Sloeber activates constantly the reset to the EPS8266 preventing the EPS to boot. And this is the reason:
Below the reset() method in your class Serial:

public void reset() {
	setDTR(false); // DTR HIGH
	setRTS(false); // RTS HIGH

	try {
		Thread.sleep(100);
	} catch (InterruptedException e) {// JABA is not going to add code
	}
	setDTR(true); // DTR LOW
	setRTS(true); // RTS LOW

}

This piece of software 'accidentally' works in a NodeMCU because looking to the timing diagram you can see that the first instruction puts DTR HIGH while RTS is LOW. This generates a reset to the EPS. Then it sets RTS HIGH resulting in releasing both, the ESP_RST and ESP_GP0. Hereafter DTR LOW resulting in a low level to ESP_GP0 and as last RTS LOW what results that both, ESP_RTS and ESP_GP0 are high again.
Simply said, this piece of code does not belong to any ESP related design.
Remark: Surprisingly, downloading firmware to the ESP8266 succeeds successful for a NodeMCU as well as the ESP in my hardware. At this moment I did not have a look to that part of code.

Recommendations for ESP8266 support in Sloeber.
I suggest to adapt your reset() method in class Serial. Besides that, I recommend to have a close look how you enter the serial download mode (if you indicate me where to find that piece of code, I can have a look with an 'electric eye').
At the bottom you see my test-bench setup with my tested reset and download proposal. This is working for the NodeMCU board as well as for ESP designs with directly connected serial lines like my design.
Probably you do but I also recommend to call reset() directly after the serial.connect().

Why is my proposal working for both hardware setups (commercial NodeMCU boards and ESP designs with directly connected RTS and DTR lines).
The only reason I can imagine why the NodeMCU board has that extra circuit between the USB-serial interface and the EPS is to prevent blocking the NodeMCU to boot when both lines, RTS and DTR, are activated (both LOW). It is working but in mine opinion it is a tricky design that can lead to many errors.
Once I understand the behavior of the electrical NodeMCU design I could have a look which situations are valid for NodeMCU and 'directly connected' designs. Most tricky/difficult part is entering the download mode. With 'directly connected' RTS and DTR lines you can precicely define in software the behaviour of these lines (that are the upper two timing diagram lines RTS and DTR of the timing diagram).
In the NodeMCU design a 'delay' created by a small capacitor together with special control of the RTS and DTR lines is needed to enter the download mode.
Based on this info I can say 'situation 1' and 'situation 2' are both usable. Basically I can't program 'situation 1' in software because there always will be an order of setting signal levels and so a delay in changing output levels of related signals. That's why I sketch 'situation 2' to clarify the order of setting the signals in software: first DTR LOW then RTS HIGH. Doing so, 'directly connected' designs will go into download mode and also NodeMCU designs will go into download mode after the 'delay'.
Controlling the reset is not that tricky. Keep line DTR HIGH and toggle RTS LOW-HIGH.

My test bench with the proposals.
///////////////////////////////////
import io.sloeber.core.api.Serial;

public class MainEntry {

public static void main(String[] args) throws InterruptedException {

	// tested with jssc-2.9.2.jar
	Serial serial = new Serial("COM6", 115200);

	serial.connect(); // WARNING: hereafter both serial pins RTS and DTR are low!!!
	
	// If RTS from USB-serial interface is directly connected to the ESP reset pin then
	// the ESP is in the reset mode and will not boot! Call resetProposal(serial) before
	// displaying serial moitor data.

	// Example: put ESP in download mode
	enterDownLoadModeProposal(serial);

	// wait to simulate download time
	System.out.println("Start firmware download...");
	Thread.sleep(5000);

	// after download reset ESP back to running mode
	resetProposal(serial);

	serial.disconnect();

	System.out.println("Einde...");
	return;
}



/**
 * Enter ESP8266 download mode working for NodeMCU panels as well as for
 * ESP hardware that have ESP pins RST and GPIO0 directly connected to the
 * USB-serial interface pins RTS and DTR
 * 
 * @param serial
 * @throws InterruptedException
 */
public static void enterDownLoadModeProposal(Serial serial) throws InterruptedException {

	System.out.println("Enter EPS8266 serial download mode...");

	// generate pulse sequense to put ESP in download mode

	serial.setDTR(false); // DTR high
	serial.setRTS(true); // RTS low 
	
	// give electronics some time
	Thread.sleep(50);

	// Important: set pins in this order!!
	serial.setDTR(true); // DTR low
	serial.setRTS(false); // RTS high

	// give electronics some time
	Thread.sleep(100);

	// done
	return;
}



/**
 * Generate ESP8266 reset working for NodeMCU panels as well as for
 * ESP hardware that have ESP pins RST and GPIO0 directly connected to the
 * USB-serial interface pins RTS and DTR
 * 
 * @param serial
 * @throws InterruptedException
 */
public static void resetProposal(Serial serial) throws InterruptedException {

	System.out.println("Reset EPS8266...");

	// be sure ESP_GPIO0 is HIGH and RTS low
	serial.setDTR(false); // DTR high
	serial.setRTS(true); // RTS low

	// give electronics some time
	Thread.sleep(100);

	// release reset and start ESP booting from flash: RTS high
	System.out.println("Release Reset to boot EPS8266 from flash...");
	serial.setRTS(false); // RTS high

	// done
	return;
}

}

///////////////////////////////////

Circuit diagram NodeMCU:

Timing diagram:

[jantje]

I'm impressed with your skills. 👍
I do not understand most of what you are writing but you have clearly put in lots of effort, so thank you. 🥇
You are clearly way more skilled at electronics than me and it is nearly impossible for me to follow what you have written, let alone understand.

I have heard of the capacitor on the reset line in Arduino world as a "must have". Not sure why I write this; maybe because that is the only thing that popped into my mind. Here is one of the many pages on this issue on the internet.
https://electronics.stackexchange.com/questions/104652/why-is-there-a-100nf-capacitor-in-the-arduino-severino-reset-line

Anyway:
As your level of electronics is way above my head; I tried to read "Recommendations for ESP8266 support in Sloeber."

I suggest to adapt your reset() method in class Serial.

The reset in serial is the reset used by the serial monitor. The reset for upload (see next point) is here
https://github.com/Sloeber/arduino-eclipse-plugin/blob/master/io.sloeber.core/src/io/sloeber/core/communication/ArduinoSerial.java#L39
Checking this I see Arduino code is now https://github.com/arduino/Arduino/blob/57a931c9c4c084057417837239ad8136f8a7b1aa/arduino-core/src/processing/app/Serial.java#L101
Apart from Arduino doing serialPort.setParams(1200, 8, SerialPort.STOPBITS_1, SerialPort.PARITY_NONE); this looks pretty much the same to me.

Besides that, I recommend to have a close look how you enter the serial download mode (if you indicate me where to find that piece of code, I can have a look with an 'electric eye').

What do you mean with " the serial download mode"? Are you talking about what is called upload in Arduino world?
If so; there are several upload ways. The usb upload is here
Triggering reset if necessary

arduino-eclipse-plugin/io.sloeber.core/src/io/sloeber/core/tools/uploaders/UploadSketchWrapper.java

Line 232 in 276915d

uploadPort = ArduinoSerial.makeArduinoUploadready(highStream, mySProject, myConfDes);

The actual upload command is run in the OS environment a bit further in the code

arduino-eclipse-plugin/io.sloeber.core/src/io/sloeber/core/tools/uploaders/UploadSketchWrapper.java

Line 253 in 276915d

if (cmdLauncher.launch(monitor, highStream, outStream, errStream) < 0)

At the bottom you see my test-bench setup with my tested reset and download proposal. This is working for the NodeMCU board as well as for ESP designs with directly connected serial lines like my design.

I'm all confused here. I thought the problem was related to Serial monitor and the code change is in reset and "serial download mode" none of these are used in the serial monitor.

Probably you do but I also recommend to call reset() directly after the serial.connect().

No Sloeber does not do so and I do not want Sloeber to do so.
Why?
When my code has been running for a while and gets into problems I may want to connect to it and look at the serial output for debugging purposes. A reset at that time would be very unhelpful and even dangerous. 😒

[Vitotje]

My problem is that I do not have any Arduino experience.
My favorite is Raspberry Pi W that I mainly use for IOT designs. But because of Corona there was a lack of Pi's and I decide to investigate the usability the ESP8266 together with the Arduino IDE needed for programming and uploading firmware.

This morning I had a quick scan on the internet and when I understand well, the base of Arduino is an Atmel micro controller chip and for firmware upload they use a dedicated chip like the ATMEGA16U2 as used in Arduino UNO Rev3 design (see attachment).

The Arduino firmware upload is different compared to the EPS firmware upload. That explains to me why I see 'remarkable code' regarding how Sloeber is interfacing the ESP.

I don't know the total architecture of Sloeber. That's why I can't say what you need to change because an 'improvement here' can generate ‘a bug elsewhere’.

Apart from Arduino doing serialPort.setParams(1200, 8, SerialPort.STOPBITS_1, SerialPort.PARITY_NONE); this looks pretty much the same to me.

Sorry to say but as far as I can see, that is completely different.
And why?
Because Arduino uses the DTR line as reset line (see attached UNO schematic pin 13 of U3) and EPS needs the RTS line for reset.

What do you mean with " the serial download mode"? Are you talking about what is called upload in Arduino world?

You are right. General speaking is ‘upload’ but I read different usages especially from Chinese origin. That's why I'm not sure which word to use.

If so; there are several upload ways. The usb upload is here Triggering reset if necessary

As previous written, until now upload to EPS works OK but because I have seen doubtful interfacing code regarding ESP, it can be wise to review again how you enter the upload mode of the ESP.
That proposal shows how you can put the ESP in upload mode, which is also valid for NodeMCU boards.

I'm all confused here. I thought the problem was related to Serial monitor and the code change is in reset and "serial download mode" none of these are used in the serial monitor.

I don't know if and when you call the serial.reset() method but that piece of code is not valid for ESP.
What I see in Sloeber is that when I click the open serial monitor button, Sloeber opens serial port to ESP but keeps the reset and GPIO0 activated (LOW). Same happens when I click the 'Restart' button. That's why the serial monitor is useless for non commercial ESP designs. As said, works by accident correctly for the NodeMCU board.

No Sloeber does not do so and I do not want Sloeber to do so.
Why?
When my code has been running for a while and gets into problems I may want to connect to it and look at the serial output for debugging purposes. A reset at that time would be very unhelpful and even dangerous. 😒

No problem for me as long as the Restart button of the serial monitor works correctly.
Besides that I thought that the serial buffer of the operating system is cleared by default at the moment you open the serial port.

arduino-uno-schematic.pdf

[jantje]

My problem is that I do not have any Arduino experience.
My favorite is Raspberry Pi W that I mainly use for IOT designs. But because of Corona there was a lack of Pi's and I decide to investigate the usability the ESP8266 together with the Arduino IDE needed for programming and uploading firmware.

We all have problem and bad life choices. 😉 Nothing to be ashamed about. As you will have found out by now pi and Arduino are far from the same thing.

This morning I had a quick scan on the internet and when I understand well, the base of Arduino is an Atmel micro controller chip and for firmware upload they use a dedicated chip like the ATMEGA16U2 as used in Arduino UNO Rev3 design (see attachment).

That is how it all started but now there is a ton of hardware that is in the Arduino world.
I look at it as follows: Before Arduino the standard way of working with programmable chips is that you decide on the chip; find all the hardware/software you need; learn all the settings and then learn how to upload. This is in my view the way to go for professionals but way to difficult for students/artists/amateurs.
Arduino starts from a board. Given the board the hardware is fixed, the software is fixed and the command options are fixed.
That way a build (called verify in Arduino world) or upload can be as simple as a click on a button.
As the used ARM chips which use a royalty free compiler this is a cheap and easy way to program a chip.
In my view that is the reason for the initial success of Arduino.

Arduino made a framework to support their set of ARM chip based boards. Over time that framework was extended to support other chips. (Teensy from Paul Stoffregen has played a big role here. )
So now there is "known name" with a framework that is pretty open to any hardware and any way op upload (like OTA, the yun, leonardo or teensy ...). And many boards "are born".
The ESP chip manufacturer provided a "plugin package" for the Arduino framework with makes it work with Arduino IDE. And they became very popular in the Arduino world due to cheap prices and wifi support.
Sloeber implements the Arduino framework on top of eclipse CDT. As Sloeber implements the Arduino framework all the boards that work in the Arduino IDE should work in Sloeber (bugs and deviations in implementation make it not 100%)
What is not part of the framework are features like the serial monitor or the plotter.

The Arduino firmware upload is different compared to the EPS firmware upload. That explains to me why I see 'remarkable code' regarding how Sloeber is interfacing the ESP.

The Sloeber behaviour should be the same as the Arduino IDE behaviour. And as explained above: the Arduino behaviour is based on a "board" not a chip.

I don't know the total architecture of Sloeber. That's why I can't say what you need to change because an 'improvement here' can generate ‘a bug elsewhere’.

Indeed. I have plenty of tests to test compile and I have some boards that I use to test upload. However serial monitor testing is not easy and It all boils down to manual tests. Thanks for understanding.

Sorry to say but as far as I can see, that is completely different.
And why?
Because Arduino uses the DTR line as reset line (see attached UNO schematic pin 13 of U3) and EPS needs the RTS line for reset.

As openport is basically connect (and as I just refound also does the set parameters https://github.com/Sloeber/arduino-eclipse-plugin/blob/master/io.sloeber.core/src/io/sloeber/core/api/Serial.java#L185) and dispose is basically close port I would say the prepare for upload is pretty much the same.
I guess you are confused because the reset you looked at originally is the reset of the serial monitor not the "reset" during "prepare for upload"
Checking what I was writing I looked at the boards.txt of the ESP8285 and I found something I had not yet seen
esp8285.menu.ResetMethod.nodemcu.upload.resetmethod=--before default_reset --after hard_reset
It looks like a "new" (for Sloeber) "reset instruction" has been introduced. I made a new issue for that one #1540
Because of this Sloeber will behave differently from Arduino IDE in regards of Upload. On first sight I would say that Sloeber does not intend to trigger a reset before nor after upload.

As previous written, until now upload to EPS works OK but because I have seen doubtful interfacing code regarding ESP, it can be wise to review again how you enter the upload mode of the ESP.

Given above "you are right". I will look into it.
Note that historically the reset before upload has been very important but reduced in importance over time.
Basically what happens is: reset + run upload command. Over time the command has included the reset because timing issues kept creeping up -which are easier to manage when you do not have to start a program in between-.

I don't know if and when you call the serial.reset() method but that piece of code is not valid for ESP.
What I see in Sloeber is that when I click the open serial monitor button, Sloeber opens serial port to ESP but keeps the reset and GPIO0 activated (LOW). Same happens when I click the 'Restart' button. That's why the serial monitor is useless for non commercial ESP designs. As said, works by accident correctly for the NodeMCU board.

While investigating I found that the setDTR and setRST simply are not taken into account.
Maybe fixing that solves your problem. I created a new issue #1541

Besides that I thought that the serial buffer of the operating system is cleared by default at the moment you open the serial port.

I would expect so but .... I'm not a java programmer 👶
This whole Serial thing is a nightmare for me (linux versus windows differences as well). I got it all to work properly one day; but it seems there has been some regression since then.

[Vitotje]

Let's wait and see.
You and your team did a great job. Sloeber is for me a very nice Eclipse plugin even with the serial monitor problem.
And remember, you will never find the last bug... 😉

[jantje]

Thanks

[jantje]

The tests for #1541 are currently running.
Please test the "new" behaviour and see if this fixes your problem.
Note that if you open the serial monitor by clicking on the toolbar when having a project that has Serial.begin in the .ino file the automatic connection to the serial port will not take into account the DTR setting in the gui.