Messages stuck in the queue

[Rel87]

I have messages stuck in the queue (C:\Inetpub\mailroot\Queue), where can I
see to whom this messages are to be delivered? (the log file doesn't gives me
the info, just the email addresses that where delivered or not)

Thanks in advance,
Ariel

[Ron Hinds]

Post by Rel87
I have messages stuck in the queue (C:\Inetpub\mailroot\Queue), where can I
see to whom this messages are to be delivered? (the log file doesn't gives me
the info, just the email addresses that where delivered or not)
Thanks in advance,
Ariel

You can use a text editor like Notepad or TextPad (http://www.textpad.com -
free & MUCH better!) to open the files and see who the intended recipient
is.

[Sanford Whiteman]

You can use a text editor like Notepad or TextPad
(http://www.textpad.com - free & MUCH better!) to open the files and
see who the intended recipient is.

Ron -- I'm not just trying to correct you at every turn here, rather
trying to prevent misinfo in the ng archives if I can -- but this is
incorrect.

The primary data stream of the file is the RFC822 message headers/body
ready to be sent as-is to the remote server. This will NOT include
envelope sender + recipient information (which is what the OP wants)
in common legitimate cases (mailing lists, BCCs), and furthermore is
readily forged.

The :PROPERTIES and :PROPERTIES-LIVE streams are where the actual
routing information is stored.

--Sandy

[Ron Hinds]

Post by Sanford Whiteman

You can use a text editor like Notepad or TextPad
(http://www.textpad.com - free & MUCH better!) to open the files and
see who the intended recipient is.

Ron -- I'm not just trying to correct you at every turn here, rather
trying to prevent misinfo in the ng archives if I can

Don't worry about it. I agree with you it's best if there is good info
available ;-)

Post by Sanford Whiteman
-- but this is
incorrect.
The primary data stream of the file is the RFC822 message headers/body
ready to be sent as-is to the remote server. This will NOT include
envelope sender + recipient information (which is what the OP wants)
in common legitimate cases (mailing lists, BCCs), and furthermore is
readily forged.
The :PROPERTIES and :PROPERTIES-LIVE streams are where the actual
routing information is stored.
--Sandy

Hmmm... I could have sworn I've used that technique before but maybe I was
thinking of messages in the Badmail folder.

[Sanford Whiteman]

Post by Ron Hinds
Hmmm... I could have sworn I've used that technique before but maybe I
was thinking of messages in the Badmail folder.

You probably were. Badmail gets split into separate plain-text messages
for each queue object.

--Sandy

[Rel87]

Post by Ron Hinds
You can use a text editor like Notepad or TextPad (http://www.textpad.com -
free & MUCH better!) to open the files and see who the intended recipient
is.

Just a quick note, Textpad is an excellent tool, but it is not free, it's
something like shareware or "try before you buy". But if you can live with
the nag screen and the moral pain, it's ok this way :)

[Sanford Whiteman]

Post by Rel87
I have messages stuck in the queue (C:\Inetpub\mailroot\Queue), where
can I see to whom this messages are to be delivered? (the log file
doesn't gives me the info, just the email addresses that where delivered
or not)

Please read that aloud to yourself: "Where can I see to whom... messages
are to be delivered? The log file [gives me] just the email addresses
that were delivered or not." ;)

That's probably not the best way to ask your question.

What you mean is: "The log file only shows me the data sent/received
during TCP/IP connections that could successfully be made with remote
servers. I want to see all the recipient info for messages left in the
queue, including those recipients for whom a TCP/IP connection could not
even be established to the domain's MX or A record."

You can view the information you want by examining the :PROPERTIES
alternate data stream of .EML messages in the queue. This is a binary
file (although the .EML's primary data stream is plain-text).

--Sandy

[Rel87]

Thanks, Sandy, that's exactly what I was looking for. Please excuse me for my
English, it is not my native language.
As you correctly guessed, it's a mailing list with BCC, so the
destination(s) doesn't appear in the EML.
I checked the :PROPERTIES data stream, and here are all the original
recipients, unfortunately it doesn't give a clear indication to which
recipients the message was delivered and to which not, must be somewhere in
the binary content in the file, I will have to do a correlation with the logs.

The problem is that I am having emails delivered many times to some of the
recipients, probably (I thought) because of slow connections with some hosts.
I checked the log file and found that effectively there was some breaks to
the SMTP dialogs that could lead to the repetitions, but I found too that
sometimes the email was delivered completely to some recipients, with
presence of the >QUIT <221 Closing connection. dialog, but anyway later the
messages to these recipients where delivered again, as if the SMTP server did
not deleted the email address from the recipients list, maybe because there
where other recipients in the same .EML that where not delivered? I don't
believe this is a bug or something in the IIS SMTP server, or I'm missing
something?
Finally I moved the messages from Queue to other folder so they will not be
delivered again, but I don't know for sure to which recipients it was
delivered, without a deep revision of the log.
I will adjust the SMTP parameters so not so many emails are delivered at the
same time, and limit the conections for each domain to 1, hope this fix the
problem, if not I will have to change to another SMTP program :(.
Any thoughts?

Thanks again,
Ariel

Post by Sanford Whiteman

Post by Rel87
I have messages stuck in the queue (C:\Inetpub\mailroot\Queue), where
can I see to whom this messages are to be delivered? (the log file
doesn't gives me the info, just the email addresses that where delivered
or not)

Please read that aloud to yourself: "Where can I see to whom... messages
are to be delivered? The log file [gives me] just the email addresses
that were delivered or not." ;)
That's probably not the best way to ask your question.
What you mean is: "The log file only shows me the data sent/received
during TCP/IP connections that could successfully be made with remote
servers. I want to see all the recipient info for messages left in the
queue, including those recipients for whom a TCP/IP connection could not
even be established to the domain's MX or A record."
You can view the information you want by examining the :PROPERTIES
alternate data stream of .EML messages in the queue. This is a binary
file (although the .EML's primary data stream is plain-text).
--Sandy

[Sanford Whiteman]

Post by Rel87
I checked the :PROPERTIES data stream, and here are all the original
recipients, unfortunately it doesn't give a clear indication to
which recipients the message was delivered and to which not, must be
somewhere in the binary content in the file, I will have to do a
correlation with the logs.

Yes, getting the most recent delivery result is tougher; I haven't
deciphered the binary format to that extent.

You'll have to do more log correlation.

Post by Rel87
...the SMTP server did not deleted the email address from the
recipients list, maybe because there where other recipients in the
same .EML that where not delivered?

Certainly, that would be (as you note) a giant bug! I doubt that's
what's happening. I imagine you've already checked to make sure the
same recipient did not get sent to multiple times and thus appears in
multiple queue objects?

Other than that, there is one rare-ish situation which can cause
non-delivery even when a QUIT command appears to be sent: some
malfunctioning MTAs disconnect after the trailing "." is sent. In
other words, they disconnect the socket while the QUIT data is still
outbound. If the socket is dropped "dirty" like that, the sending
server may retry. But that is, as I said, rare and should not occur in
any quantity (unless perhaps you have local network issues which are
accelerating its frequency?).

Post by Rel87
I will adjust the SMTP parameters so not so many emails are
delivered at the same time, and limit the conections for each domain
to 1, hope this fix the problem...

That can only help your situation, certainly. It is possible that you
are saturating your line. I don't know about the volume of messages
you are sending and the speed of your upstream, but it is remarkably
easy to saturate a T1 with an unrestricted, multithreaded mail blast.
The more dropped packets you have, the more IIS' simple
application-level logs will fail to reflect what is really happening
on the wire. You should get a packet sniffer on there and check for
such errors.

I can say with surety that IIS SMTP is an extremely powerful MTA that
can send millions of messages a day on proper hardware, so I doubt
your problem starts within IIS.

--Sandy

[Rel87]

Post by Sanford Whiteman
Certainly, that would be (as you note) a giant bug! I doubt that's
what's happening. I imagine you've already checked to make sure the
same recipient did not get sent to multiple times and thus appears in
multiple queue objects?

Yes, I've already checked that.

Post by Sanford Whiteman
Other than that, there is one rare-ish situation which can cause
non-delivery even when a QUIT command appears to be sent: some
malfunctioning MTAs disconnect after the trailing "." is sent. In
other words, they disconnect the socket while the QUIT data is still
outbound. If the socket is dropped "dirty" like that, the sending
server may retry. But that is, as I said, rare and should not occur in
any quantity (unless perhaps you have local network issues which are
accelerating its frequency?).

This seems not to be the case, since I get the "221 Closing connection."
answer from the remote server.

Post by Sanford Whiteman
That can only help your situation, certainly. It is possible that you
are saturating your line. I don't know about the volume of messages
you are sending and the speed of your upstream, but it is remarkably
easy to saturate a T1 with an unrestricted, multithreaded mail blast.
The more dropped packets you have, the more IIS' simple
application-level logs will fail to reflect what is really happening
on the wire. You should get a packet sniffer on there and check for
such errors.

Thanks for all the recomendations, according to my calculations the upstream
pipe should be enough, but we have some slowness with some destinations that
by coincidence are the destination of a lot of the emails in the mailing. :(
Will wait for the next mailing (it's only every 2 months, so I don't have
many chances to study the matter, but the receivers are very important
persons in my organization) to see if the restrictions in the SMTP properties
I've put does the magic.

Thanks again,
Ariel

[Sanford Whiteman]

Post by Rel87
Will wait for the next mailing (it's only every 2 months, so I don't have
many chances to study the matter, but the receivers are very important
persons in my organization) to see if the restrictions in the SMTP
properties I've put does the magic.

OK. Please get in touch with me off-list just before you're going to send
the blast. I'd be interested to take a look while the problem is
occurring.

--Sandy

[Udo]

Hi,
I got some similar problem:
Is there any way of seeing the BCC recipients while a message is still in the queue directory?
Here I can only see the .eml file, but where is the recipient information stored? The IIS must expand all the BCCs somewhere in order to "know" where to send the message to after all...

Thank you very much for your help in advance!
Udo

Posted via DevelopmentNow.com Group
http://www.developmentnow.com

[Sanford Whiteman]

Post by Udo
Is there any way of seeing the BCC recipients while a message is still
in the queue directory?

Search my posts for PROPERTIES-LIVE.

--Sandy

[Udo]

Hi Sandy,

thanks for your feedback. I've seen that you were talking about these properties and live streams, but where can I find them?
Are they files or what else?
Also in the 'queue' directory I only have these eml files and no splitted files as in the 'bad' directory. And I can't find anything else anywhere in the 'mailroot' dir.

Thank you very much for clarifying this
Udo

Posted via DevelopmentNow.com Group
http://www.developmentnow.com

[Sanford Whiteman]

Post by Udo
thanks for your feedback. I've seen that you were talking about these
properties and live streams, but where can I find them?
Are they files or what else?

They are alternate data streams within the queue files.

Post by Udo
Also in the 'queue' directory I only have these eml files and no
splitted files as in the 'bad' directory. And I can't find anything else
anywhere in the 'mailroot' dir.

That's right: the queue file bodies, SMTP envelope info, and control
metadata are all unified in one file through the of ADS.

Some applications are incapable of reading ADS in some contexts. At
simplest, go to a command prompt, change to your queue folder, and try
this:

more < testfile.eml:PROPERTIES-LIVE

Also get the streams.exe tool from MS.

--Sandy

[Udo]

Meanwhile I've googled for these PROPERTIES-LIVE streams and found some more hints, but for example 'notepad.exe anyemail.eml:PROPERTIES-LIVE' results in a 'file not found' message.
I'm using an IIS 5.0 on an NTFS partition with Win2K as member server WITHOUT ActiveDirectory.
Do I maybe need Active Directory or a later IIS version to be able to see the LIVE streams?
But even if this would be the case, my IIS would have to store the recipients' addresses nevertheless, so there should still be a way of finding them out?

Udo


Posted via DevelopmentNow.com Group
http://www.developmentnow.com

[Sanford Whiteman]

Post by Udo
Do I maybe need Active Directory or a later IIS version to be able to see the LIVE streams?

No, that's not relevant.

NTFS alternate data streams have nothing to do with AD.

--Sandy

[Udo]

Hi Sandy,

thank you very much, this helped!
I looked deeper into this ADS thing and got the intention behind it and also the sysinternals utility, MS StrmExt.dll for seeing the ADSs as an explorer shell extension and everything else I need.

One last question for my understanding:
These streams must be stored somewhere by the OS. Where exactly? In a special file/in the registry/some database?
So is it possible to delete all streams on a harddisk in one go by deleting the according resource?

Thank you very much again foryour kind help!

Udo

Posted via DevelopmentNow.com Group
http://www.developmentnow.com

[Sanford Whiteman]

These streams must be stored somewhere by the OS. Where exactly? In
a special file/in the registry/some database?

They are a feature of NTFS and stored within the filesystem and its
indices, just like default (a.k.a. primary) data streams.

You might look at ADSes as being the opposite of hardlinks.

While hardlinks allow you to refer to a single underlying set of bytes
by two different names, ADSes allow you to use a single
name-plus-subname to refer to two different, but usually very closely
related, sets of data. Each data stream is equivalent to a fully
separate "file" in conventional terms. Indeed, one file's alternate
data stream can be copied out into another standard, standalone file
-- becoming that second file's default data stream.

So is it possible to delete all streams on a harddisk in one go by
deleting the according resource?

Not that I know of, no.

Converting an NTFS partition to a FAT partition and back again will
remove ADSes, but it will remove a heckuva lot of other stuff as well,
like NTFS permissions. In theory, you could backup NTFS permissions to
an answer file, convert the filesystem, convert it back, and reapply
permissions. Windows would just recreate any ADSes as needed. But don't
do this :)

--Sandy

[Udo]

Thanks for your thorough explanation, Sandy!

Still one thing is not answered yet:

Where exactly can I find these alternative data streams 'physically'?
You said they are stored like standard streams, that means like any standard file.
But I haven't found any hint where these alternative strems' files are located.

I assume there must be a file for every additional stream that must be locatable in the file system just like any standard stream has a file that can be found in its directory, viewed, edited etc.

Is this the case?

Thank you again

Udo


Posted via DevelopmentNow.com Group
http://www.developmentnow.com

[Sanford Whiteman]

Post by Udo
I assume there must be a file for every additional stream that must be
locatable in the file system just like any standard stream has a file
that can be found in its directory, viewed, edited etc.

No, there is no "physical" file that can be viewed with non-ADS-aware
utilities.

Of course, there is no such thing as a physical file for either primary or
alternate streams -- just pointers to sets of disk blocks that are read
together _as if_ they have a permanent start and end point. The structure
of a primary or alternate stream, once it gets loaded into memory, is
exactly the same. You can store a database in an alternate stream, with
an INI file in the primary stream of the same file; or vice versa. Only
thing you can't do is access the alternate streams from applications that
aren't aware of the possibility of ADSes. For example, an application
that prohibits a colon in a filename will choke if you try to enter an
ADS, since an ADS is located using a colon as the separator.

--Sandy

[Udo]

Post by Sanford Whiteman
No, there is no "physical" file that can be viewed with non-ADS-aware
utilities.

I understand that non ADS.aware applications can not view the streams' files, but for apps that *are* ADS aware: Can they see the streams as files if they have the ability built in?

These data streams must be stored somewhere after all...
For example, if I have a 1GB HD and I fill 900MB of it with an alternate data stream, I can only have 100MB left, but in what form of data do the 900MB appear to for example checkdisk as a non ADS aware application?
And is it possible to see the 900MB file for an ADS aware application as a real file that can be located in an 'alternate directory' or something?
For my 'classical file oriented :-)' understanding all data, alternate or not, should be locatable somewhere as a file or something else from an ADS aware application.
Or is this classic view on these things the wrong approach for ADS?

Sorry to go deeper into this issue, but this is really something that nobody could make clear so far, even not all the literature I found for it was making this detail clear.

Udo


Posted via DevelopmentNow.com Group
http://www.developmentnow.com

[Sanford Whiteman]

I understand that non ADS.aware applications can not view the
streams' files, but for apps that *are* ADS aware: Can they see the
streams as files if they have the ability built in?

As files without any more qualification, no.

They would always be referenced using either the colon-separated
format 'name:stream' (typically if you are passing a stream name to
compiled application, or if you already know the name of a stream) or
by using special API functions (if you are scanning for unknown
streams). The functions are able to extract all of the stream names
and stream data given the standard name of the file (first you get a
handle to the file by name, _then_ you loop through its streams,
reinforcing the parent-child kind of relationship I've been trying to
put across).

These data streams must be stored somewhere after all... For
example, if I have a 1GB HD and I fill 900MB of it with an alternate
data stream, I can only have 100MB left, but in what form of data do
the 900MB appear to for example checkdisk as a non ADS aware
application?

Chkdsk *will not* increment the number of files if you add ADSes to a
given file. It "will" see that there are more allocation units in use
on the disk.

And is it possible to see the 900MB file for an ADS aware
application as a real file that can be located in an 'alternate
directory' or something?

No.

An ADS-aware application must still treat ADSes as ADSes. They are not
simple files. Either the colon-separated syntax or the ADS-aware API
functions must be used.

For my 'classical file oriented :-)' understanding all data,
alternate or not, should be locatable somewhere as a file or
something else from an ADS aware application. Or is this classic
view on these things the wrong approach for ADS?

Wherever your classic view is taking you is obvs. causing too much
confusion for a pretty simple, albeit cool, technology. I think your
view is too rooted in the (mis)apprehension that discrete files are
"real" or "physical" -- as opposed to being arbitrary constructs of
the particular filesystem in use that allow assemblages of bytes to be
viewed as discrete wholes. Try accessing an NTFS partition as if it
were FAT32; you'll see that _all_ your files are suddenly un-"real",
not just the ADSes!

ADSes are definitely present on disk in physically separate (and quite
possibly discontiguous) areas, just like primary data streams. You can
see that they take up allocation units, just like anything else that
gets saved. Major, really only, diff is that they don't have their own
standalone names, being only addressable as subnames of a "master"
file. This ensures that ADSes, which typically provide supporting
data, such as user settings, or store state, as with the SMTP queue,
will be deleted along with the (sole) data to which they relate.

--Sandy

[Udo]

Thank you very much, Sandy for giving me (and I'm sure many others also) this deep insight into this kind of 'dark matter', at least seen from the classical 'file related' view that most people will surely have gotten used to over the last 20 years.

I appreciate this very much, thanks again!!

Udo

Posted via DevelopmentNow.com Group
http://www.developmentnow.com

[Eduardo Coasaca]

i've been looking answers for the same problem, i get "Messages stuck in the
queue" and while i was trying i notice something that could be the solution
(centainly not the professional answer):

1. click start, control panel, administrative tools, IIS services
2. expand PC and right click "default virtual SMTP server"
3. select delivery tab (spanish: ENTREGA)
4. set lower values for: "first retry interval", "second try interval",
"third retry interval", "subsequent retry interval"... (i set them to 1)

now it's working fine for and hope this to be usefull