Module telnet behavior

#1

After weeks of scratching my head over inconsistent telnet behavior, I think I’ve finally isolated what causes my problem, though I have no idea why or how to fix it.

WIZ145SR running 1.6.1 firmware

Serial side: a small microprocessor console port, 9600-N-8-1.

Network side: telnet from a Red Hat Enterprise Linux Server release 5.7 (Tikanga) box

Scenario 1:
When I establish the telnet session before allowing the microprocessor to boot (i.e. don’t push any characters out the console), everything works as it should as long as I make sure the telnet session is in character mode instead of the default line mode.

Scenario 2:
When I allow the microprocessor to boot and push a bunch of characters out the console before establishing the telnet session, all kinds of weird things happen. 1) I see a bunch of stuff come out of the terminal server that it must have received before the telnet session was established. 2) I see the terminal server throw some garbage character or characters toward the console. 3) all characters are now echoed.

When I start in scenario 1 (and all is good), terminate the telnet session gracefully, and then re-establish the telnet session, I’m back to scenario 2. It looks like re-establishing the telnet session throws some junk at the console that gives my micro fits.

Why would the terminal server appear to be buffering characters from before a telnet session was established? Wouldn’t initializing the session clear any characters in the buffer? I’m not talking about characters coming in during the initialization. The characters I see were sent out the console well before I attempt to establish the telnet session.

Why would establishing a telnet session force any character or characters out the serial port?

#2

Anyone out there? This is a show stopper for me. I’ve got 100 modules here I can’t use if I don’t find a solution/work around for this.

Gary

#3

Hi Gary,

I’m sorry for our late response.
I’m the developer of WIZ145SR.
I’ll support you since now.
To answer you correctly, I have to check it at my side.
Please wait for several days and I’ll give you my feedback as soon as possible.

Thank you.

James.

#4

Hi Gary,

I tested WIZ145SR with firmware ver 1.6.1 and figured out what you told.
I looked through the source code of WIZ145SR carefully and found out what made this symptom.
Yes, it has had some bug in handling serial buffer.
There are several timer for handling serial data and several pointer to serial buffer.
When WIZ145SR receives data after connection is established, initialization of several timers and pointer is done correctly.
But the initialization is done by receiving data from serial port and timers and pointer is already being updated, even though the tcp connection is not established yet.
This is the reason why WIZ145SR pushed wrong data.

I fixed and tested it at my side.
I hope you to test with my revised firmware at your side.
WIZ140v1_6_2_140326.zip (24.5 KB)
I put its version 1.6.2 temporarily and it is not the official version yet.

In order to release its firmware officially, we need more time to test by our rule.

Anyway, you can test with the attached file at your side.(I compressed it as a zip file. please decompress it first.^^)

Thank you.

BR, James.

#5

James,

Thanks for looking at this.

I think things are improving. I don’t see the buffered characters when I start a telnet session now, but I do still see some characters thrown out the serial port when a telnet session is started. I don’t think this is supposed to happen.

Output from the terminal server serial port when I start a telnet session:
0xFF
0xFD
0x03
0xFF
0xFD
0x01
~2.5ms gap
0xFF
0xFC
0x3c
~32ms gap

Baud rate and signalling looks correct, but establishing a telnet session shouldn’t initiate anything out the serial port.

#6

Hi gthompson,

I’m happy to hear that your test is going forward.
Anyway I’m sorry you’re still in some trouble.

When I was testing with the firmware ver 1.6.2 which I uploaded here, I couldn’t see the symptom you said above.
I’ll test again. Could you tell me your test condition more details? How are all devices connected?

When were above data, which were displayed on telnet program, sent from your terminal server?
And are those real data from terminal server or garbage?
If those are real data, I think timing mismatch between serial and ethernet make outputting data on telnet program.
Because serial port is much slow compared to ethernet, your terminal server may be still having data to output into serial port in its tx buffer when telnet session is created.
So, I recommend you check the timing of those signals, RXD of UARTs and STATUS pins.
When every connection is established, its corresponding STATUS pin is changed to LOW.
If RXD is not idle after STATUS pin is changed, above data is correct.
If there is no transition of RXD after changing of STATUS pin, my firmware has still some fault.

Please inform me your test result.

Thank you.

James.

#7

That was real output from the terminal server taken on an oscilloscope, so I know the timing and protocol look correct. This output occurs immediately after I establish a telnet session with the module, but before I start to send it any real data.

It is consistent in that I get the same data every time, and every time I start a new telnet session, I get the same data.

I see no difference between 1.6.1 and 1.6.2 in this regard.

Here is my configuration:
network side attached to a live network
I telnet into the terminal server from a Linux box
Linux version 2.6.18-274.18.1.el5 (mockbuild@x86-001.build.bos.redhat.com) (gcc version 4.1.2 20080704 (Red Hat 4.1.2-51)) #1 SMP Fri Jan 20 15:11:18 EST 2012

Serial port 0 is configured for 9600,N,8,1 port 5000.
Serial port 0 is connected to a small microprocessor running Linux u-boot. Only RXD/TXD are connected. Handshake signals are not used. The microprocessor is booted to the u-boot prompt and sitting waiting for a command. It will not send any characters to the terminal server unless it receives something.

So, sequence of events is as follows:

  1. power on terminal server and target system
  2. bring target system out of reset
  3. wait for target system to boot
  4. from linux box> telnet axx-g004-ts 5000
  5. I see junk start to come out of the terminal server on the UART1_TX pin (module J1 pin 7).
  6. target system sees this junk, echoes it back to the terminal server and runs off into never-never land.

If I move step 2 after step 4, everything works as it should (the target isn’t alive to see the junk at the time the telnet session is started, so it is happy). Unfortunately, this is not always a viable option since I need to be able to initiate and terminate telnet sessions without resetting the target.

If I remove the Wiznet module from the system, and wire the target console connection to a Cisco 2800 or a Digi PortServer TS16 terminal server, everything works great. I can initiate and terminate telnet sessions at will without disturbing the target. Unfortunately, these don’t fit in my chassis very well posting.php?mode=reply&f=21&t=667#

I haven’t looked at what is happening on UART1_RTS or UART1_CTS yet, but I will.

Gary

#8

OK, a little more data.

When I start the telnet session in your default line mode setting, I don’s see any characters out when the telnet session is initiated. When I escape back to the telnet prompt and change the mode to character, I see the above characters come out the serial port. Similarly, when I start a telnet session with a .telnetsrc file that automatically switches to character mode, I see the characters.

So I guess I see why you don’t see the same thing I do. Try changing the telnet mode from line mode to character mode and see if you don’t see the same extraneous character pop out the serial port.

BTW, I looked at CTS and RTS, and they both stay at 0 (on the module pins) through the whole process. I have an LED on the STATUS pin, and I do see it light (STATUS goes low) well before the characters are sent.

Gary

#9

OK, I understood what the problem was.

At first, WIZ145SR doesn’t have any special consideration for telnet modes.
It just exchage data transparently between serial port and ethernet socket.
I think normal telnet server do some different action according to telnet mode.
This is why our WIZ145SR didn’t operate as you expected.
I’ll check what I should do to meet your demand. Please wait for a few days.

And as I told you before, You found that WIZ145SR get received data via UART port from your serial device after telnet session was established.
The timing of signal transition is the evidence of what I told you.
Yes, as CTS and RTS is signals for flow control, if you didn’t set RTX/CST mode for flow control, those wouldn’t change those level.
When STATUS pin went to low, it means telnet session was established and WIZ145SR send all data received via UART from this point. WIZ145SR worked well even at your side as I expected.

Anyway, the problem which you encountered came from the lack of special consideration for telnet mode.
I’ll survey how I can fix it and inform you the result as soon as possible.

Thank you.

James.

#10

Hello Gary,

I did some tests to reproduce the behavior. My Testsetup was Putty as Telnet client and hterm as serial client programme.
When I set putty to use Telnet mode I can see the extra characters. When I set putty to raw mode there are no extra characters. As James already mentioned this is because the current firmware has no support for “telnet” mode but only for “raw” mode.

So my question is, do you really need the telnet mode? Or is “raw” mode sufficient? If you set your client to “raw” mode it will immediately work as you expect it. But if you really need telnet mode, then James has to enhance the firmware for this.

Please tell if raw mode works for you. You can test this easily this with putty as client.

Regards Tobias

#11

Tobias,

Thanks for your help with this.

Unfortunately, I really need telnet mode. I’ve got a bunch of software developers and hardware validation guys using these terminal servers to access test systems. They’ve got years worth of automated test scripts and processes that I can’t really ask them to change. My decision to transfer from rack mounted Cisco terminal servers to Wiznet embedded terminal servers needs to be completely transparent to them.

If I had realized your terminal servers weren’t really telnet-capable, I would have never gone down this path.

#12

gthompson,

OK, I’ll try to add minimal telnet capability onto WIZ145SR.
Implementing full function for telnet mode may take long time.
So, I try to change my code not to bypass “Telnet Negotiation Message” to Serial port.
Can you test with it?
Or do you need some more function?

And regarding to implementing full function for telnet mode, I need to consider carefully and I’ll discuss my colleagues.

Thank you.

James.

#13

Ideally, we’d like the telnet negotiation to work, but we can live without it by forcing our end through .telnetrc.

I think not passing the telnet negotiation string on to the serial port is all we really need for now. I’d be glad to test it once you are ready.

Thanks to everyone for helping us get this going!

Gary

#14

Hi Gary,

I finished firmware and configuration tool customization to support telnet mode.
Now my colleagues and I are doing internal test.
After test, I will post here or email to you with the firmware and configuration tool.

Please wait for a few days.

Thank you.

James.

#15

Thank you, James. I’ll look for your updates.

Gary

#16

Hi Gary,

I posted new firmware and configuration tool here, [Notice]Latest FW and Configuration Tool
Please download them and test with them on you side.

Thank you.

James.

#17

James,

I finally got some quality time with your new release, and I’m still seeing something strange.

I made some comparisons between a Cisco terminal server that works for us and the Wiznet module. Using telnet netdata and termdata debug commands, I was able to comapre what was happening on both the netwkr side and ther terminal side as far as telnet was concerned. THEY BOTH LOOK IDENTICAL!

Cisco

network data ()
=> > 0x0 0d00
< 0x0 0d0a3d3e20

=> > 0x0 03

network data (CTRL-J)
=> > 0x0 0a
< 0x0 0d0a3d3e20


terminal data ()
< 0x0 0d

=> > 0x0 0d0a3d3e20

terminal data (CTRL-J)
< 0x0 0a

=> > 0x0 0d0a3d3e20

Wiznet

network data ()

0x0 0d00
< 0x0 0d
< 0x0 0a

< 0x0 3d
=< 0x0 3e

< 0x0 20

network data (CTRL-J)

0x0 0a
< 0x0 0d
< 0x0 0a

< 0x0 3d
=< 0x0 3e

< 0x0 20


terminal data ()
< 0x0 0d

0x0 0d

0x0 0a
=> 0x0 3d

0x0 3e
0x0 20

terminal data (CTRL-J)
< 0x0 0a

0x0 0d

0x0 0a
=> 0x0 3d

0x0 3e
0x0 20

This is definitely NOT what I expected. If they really are identical, both should work equally well, right?

Unfortunately, while telnet reports a lone 0x0d () on the terminal side, what I actually get on the scope is 0x0d 0x00 ( NUL). Where did that null come from?

It looks like a standard Linux u-boot will accept a CRLF, a CR, or a LF as suitable end of line characters. It doesn’t know what to do with the NUL, though, so it ends up stuck in the buffer until the next command comes along, causing the next command to parse incorrectly.

It would be interesting to see if you see the same thing. When you send just a 0x0d from the network side, what comes out the serial side?

Gary

#18

OK, I’ll try to test it soon.
Thank you for your feedback.

James.

#19

Hi Gary,

I’m sorry I couldn’t produce its symptom on my side.
It may be a reason that data from terminal was separate into several block.
If you set your module up like below picture, you would get the whole data from terminal at once.

Can you give me a feedback after you test again with above setting?

Thank you.

BR, James.

#20

What does changing the Time field to 100 do? I don’t really see any difference.

Make sure you have your telnet session configured as I do in single character mode. I’ve got “mode char” set in my .telnetrc file to force character mode on all telnet sessions. If I stay in the line mode your telnet defaults to, I get , on the serial side, which u-boot interprets as a double end of line.

Even with Time=100, I see a from the remote terminal pass a , to the serial port.

telnet> status
Connected to axx-g004-ts.
Operating in single character mode
Catching signals locally
Remote character echo
Escape character is ‘^]’.

telnet> display
will flush output when sending interrupt characters.
won’t send interrupt characters in urgent mode.
won’t skip reading of ~/.telnetrc file.
won’t map carriage return on output.
will recognize certain control characters.
won’t turn on socket level debugging.
won’t print hexadecimal representation of network traffic.
won’t print user readable output for “netdata”.
won’t show option processing.
won’t print hexadecimal representation of terminal traffic.

echo [^E]
escape [^]]
rlogin [off]
tracefile "(standard output)"
flushoutput [^O]
interrupt [^C]
quit [^]
eof [^D]
erase [^?]
kill [^U]
lnext [^V]
susp [^Z]
reprint [^R]
worderase [^W]
start [^Q]
stop [^S]
forw1 [off]
forw2 [off]
ayt [^T]
resp DO_DONT ECHO: 1
want DO ECHO
resp DO_DONT SUPPRESS GO AHEAD: 1
want DO SUPPRESS GO AHEAD

=>

I still see this:

changing to look at network side data

telnet> set netdata
Will print hexadecimal representation of network traffic.

this is in response to a on the remote terminal

0x0 0d00 # this is the network data from remote terminal to Wiznet
< 0x0 0d0a3d3e20 # this is the network data from wiznet to the remote terminal

=>

changing to look at serial side data

telnet> unset netdata
Won’t print hexadecimal representation of network traffic.
telnet> set termdata
Will print hexadecimal representation of terminal traffic.

this is in response to a on the remote terminal

< 0x0 0d # this is the network data from remote terminal to Wiznet
=> > 0x0 0d0a3d3e20 # this is the network data from wiznet to the remote terminal

So I see telnet reporting the serial side data as a only, but what I see on the scope is a . The is valid on the network side, but I have not found ANY other terminal server that passes that on to the serial side. If I could just get rid of the serial side I’d be a happy man.

I can get everything to work exactly as I want, as long as I remember to end all lines with a instead of a . The keyboard short cut just sends a .

changing to look at network side data

telnet> set netdata
Will print hexadecimal representation of network traffic.

this is in response to a on the remote terminal

0x0 0a # this is the network data from remote terminal to Wiznet
< 0x0 0d0a3d3e20 # this is the network data from wiznet to the remote terminal

=>

changing to look at serial side data

telnet> unset netdata
Won’t print hexadecimal representation of network traffic.
telnet> set termdata
Will print hexadecimal representation of terminal traffic.

this is in response to a on the remote terminal

< 0x0 0a # this is the network data from remote terminal to Wiznet
=> > 0x0 0d0a3d3e20 # this is the network data from wiznet to the remote terminal

I did try PuTTY and Microsoft telnet just to see if I could find a combination that worked. Both essentially do the same thing. By default, they send on the network side that you pass on as on the serial side, resulting in double end of lines to u-boot. Not catastrophic, but not great.