Welcome to MSDN Blogs Sign
in | Join
| Help

Windows CE Base Team
Blog

Windows CE kernel and storage technologies and system
tools.

RSS 2.0

Atom 1.0

Home

Email

Search
Go

Tags

Author: Andrew Rogers

Author: Ariane Jansen

Author: Brent Bishop

Author: Chaitanya Raje

Author: Dave Edson

Author: David Liao

Author: Greg Hogdal

Author: Jeremy Cooke

Author: Jim Causey

Author: Jim Muliawan

Author: Konstantin Meleshenko

Author: Kurt Kennett

Author: Matt Anfang

Author: Nachiket Acharya

Author: Oguz Sinanoglu

Author: Riki June

Author: Russ Keldorph

Author: Sankar Ramasubbu

Author: Sha Viswanathan

Author: Shilpi Gautam

Author: Sue Loh

Author: Travis Hobrla

Author: Upender R. Sandadi

Author: Vadim Yushprakh

Author: Wes Barcalow

Autor: Javier Flores Assad

BSP/Drivers

Build and Compilers

Debugging and Performance Tools

Developer Documentation

File System

Kernel

News & Misc

Platform Builder Automation

Platform Builder IDE

Registry & Databases

Remote Tools Framework Dave Edson

Testing

News

These postings
are provided "AS IS" with no warranties, and confer no rights.

Archives

August 2008
(1)

June 2008
(5)

May 2008
(1)

April 2008
(1)

March 2008
(1)

February 2008
(3)

January 2008
(1)

November 2007
(2)

August 2007
(1)

June 2007
(4)

May 2007
(5)

April 2007
(7)

March 2007
(11)

February 2007
(5)

January 2007
(13)

December 2006
(5)

November 2006
(16)

October 2006
(4)

September
2006 (6)

July 2006
(1)

June 2006
(6)

May 2006
(5)

April 2006
(5)

March 2006
(4)

February 2006
(6)

January 2006
(4)

December 2005
(2)

November 2005
(4)

September
2005 (9)

August 2005
(8)

June 2005
(6)

Events

Other
Events

Other Windows CE Blogs

Mike Hall

Windows CE Networking Team

Windows Mobile Team

TestEmbedded

HoppeRx - Debugging

Nino.Mobile

The Oswald Project

The Moth - .NET CF

Chris Tacke

Sue Loh

Other
Windows Embedded Blogs

Windows Mobile Shell Team

.NET Compact Framework Team

Steven Pratschner - .NET CF Perf

Visual Studio for Devices Team

Jason Browne

Windows CE Multimedia Team

Outlook Mobile Team

Reed & Steve

User Groups

EmbeddedFlorida

Mobile &
Embedded Developers UK

BostonPocketPC

Windows CE Resources

Windows CE
Home

MSDN chat schedule

Windows
CE Online Tutorials (they are cool, check 'em out!)

Windows
CE Newsgroups

Download
Windows Embedded Evaluation Edition

Windows
CE Shared Source Program

Webcasts
and NetSeminars

MSDN
online

Wednesday, March 15, 2006 1:57 PM ce_base

How to Programmatically Improve File System Throughput

Writing large amounts of data to
persistent media can take large amounts of time. While Windows CE does have
hardware limitations in comparison to the desktop, there are coding practices
that will increase your throughput when writing to disk/flash.  Higher throughput results in a better
overall experience with WindowsCE and applications for WindowsCE.  I will use flash media as my primary
example in this blog.

·
Know your block size.

The most important step to
programmatically increase your throughput is to know your flash block size, and
to write in multiples of the block size.
The flash block size is the smallest buffer that will be written to, or
read from, flash.  Writing less than
512 bytes at a time will take the same amount of time as writing a full 512
bytes.  If a program is writing 4
bytes at a time in random places in a file, then the throughput could be 128
times worse than another program that is buffering and writing out 512
bytes.  There is a small amount of
caching in most file systems, as well as block drivers, which may save an
application with 4-byte-write behavior from coming to a near-standstill, but
this is not guaranteed.

Buffering to a multiple of the
block size will further increase your throughput.  If you make one large write of ten times
the block size, then your program only needs to verify buffers, traverse the
file system stack, and thunk into the kernel one time to flush all the
data.  The time taken for each step
in the process of getting data written to disk is small; however, every
millisecond starts to noticeably add to overall time when there are hundreds or
thousands of writes.

Another reason to write multiples
of the block size is increasing the throughput for compression and/or
encryption.  Compression and
encryption are usually CPU-heavy, block-based algorithms that require a set
amount of bytes(usually a block multiple) to be read for every single read or
write.  Every time a byte is
written, the entire block associated with that byte must be read, the block must
be decrypted, the block must be decompressed, the byte changed, the block
compressed, the block encrypted, and finally the block will be written out.

The 4-byte-write application will
likely see their throughput slow to a crawl in this situation, as every one of
the CPU-heavy algorithms will be called 128 times for writes amounting to
4kB.  This will also wear the flash
media down considerably faster.  In
contrast, a program that is writing 16kB buffers on a media with a 4kB block
size will read in data once, and decrypt/decompress/compress/encrypt will be
called a total of 4 times for the entire operation.

In order to discover the block
size of your current flash media you will need to call CeGetVolumeInfo:

WINBASEAPI
BOOL CeGetVolumeInfo(

LPCWSTR pszRootPath,

CE_VOLUME_INFO_LEVEL
InfoLevel,

LPCE_VOLUME_INFO
lpVolumeInfo

);

(From: http://msdn.microsoft.com/library/default.asp?url=/library/en-us/wcedata5/html/wce50lrfCeGetVolumeInfo.asp
)

The first variable, pszRootPath,
can be any valid path name and the returned information will reflect the file
system which manages that path.

The structure returned
(LPCE_VOLUME_INFO) is a pointer to the following structure:

typedef struct _CE_VOLUME_INFO{

DWORD cbSize;

DWORD dwAttributes;

DWORD dwFlags;

DWORD dwBlockSize;

TCHAR szStoreName[STORENAMESIZE];

TCHAR szPartitionName[PARTITIONNAMESIZE];

} CE_VOLUME_INFO, *PCE_VOLUME_INFO, *LPCE_VOLUME_INFO;

(From: http://msdn.microsoft.com/library/default.asp?url=/library/en-us/wcedata5/html/wce50lrfce_volume_info.asp
)

dwBlockSize is the block
size of your flash in bytes.

·
Memory Mapped files for caching

If your application does not need
to flush information immediately, or can survive power loss with some data loss,
then another option for increased throughput is memory mapping the file that
you’re writing to.  The
memory-mapped file will act as a file cache, until you write it out. When the
file is written out, you get the benefit of the file being written in large
chunks, in a contiguous manner; although, the contiguousness is not as important
for flash.  The memory mapping
functions are located here: http://msdn.microsoft.com/library/default.asp?url=/library/en-us/wceobjst/html/cerefCreateFileMapping.asp

·
Use sub-directories (FATFS)

FATFS is the primary file system
used on flash media, so keeping the number of files in a particular directory
will help most devices.  The reason
the number of files in a directory is important rises from the fact that
directory entries are stored as a linear array, which must be walked on every
path-based API. When you have 15,000 files in one directory, every
open/close/move/delete/findfirst file call requires all the files in the
directory to be enumerated, which means the entire array is walked until your
file is found.  Needless to say,
this is very slow.  Saving all your
project files in one directory may seem like an easy-access solution for your
program’s enumeration, but it is very difficult for the file system to keep
up.  You can hash the filenames, or
bucket them in alphabetical order; your ordering scheme doesn’t matter as long
as the buckets are as small as the array you want walked on every path-based
file operation. If your program requires all the files be stored in one
directory, perhaps you can write a file system filter that will hide the
subdirectories for you.

·
Expensive APIS: FlushFileBuffers(), RegFlushKey(),
FlushViewOfFile()

WindowsCE is used in environments
where power loss is much more likely than the desktop.  A phone is dropped, a battery dies,
etc.  The architects, developers,
testers; everyone is aware of these scenarios and the shell, file system
manager, and OS are written with power loss in mind.  There is a lot of flushing already being
done on your behalf, which makes these functions unnecessary time-wasters for
most applications.  Sue's post: http://blogs.msdn.com/ce_base/archive/2005/09/26/474069.aspx also
adds perspective to whether application developers should be flushing.

FlushFileBuffers(): This function
will flush everything in the write-back cache, as it does not know what part of
the cache belongs to your file.
This can take a lot of time, depending on the cache size and the speed of
the media.  How necessary is
it?  There is a thread which goes
through and writes out dirty pages, so it is likely not very necessary.

RegFlushKey():  As with FlushFileBuffers(), this API
flushes a lot more than you would think: depending on the situation, RegFlushKey
could end up flushing a lot of data, up to the entire registry.  Double and triple checking that your
changed registry key has successfully been stored by using RegFlushKey() is
usually not necessary.  Ask
yourself: “Will the machine still be able to boot if this registry key isn’t
immediately saved?”  If the answer
is yes, then you probably do not need to use this API in the middle of your
process.  As with files, there is a
thread that will wake up and flush the registry on your behalf.  If you must flush your registry key, try
to do it at the end of your program’s execution, preferably at close.  This way you get all the changes that
you have made (and anyone else has made) written out.

FlushViewOfFile():  This API is useful when you are writing
to a memory mapped file, and using it as a buffering scheme.  What is not useful is flushing the view
on every 4 byte write.  If you’re
flushing for every write you make, then what is the purpose of using memory
mapped file?  Use the map as a
buffering tool to increase the amount of data changed, rather than a safety net
guaranteeing that everything will be written out precisely.  Call this only when you’re done writing
or at preset, block-multiple intervals.
Again, there is a thread that intermittently will write dirty pages to
the file, so it may not be necessary at all.

Filed under:
File System, Author: Ariane Jansen

Comments

#
What's A &amp;quot;Compaction Thread&amp;quot;?

Thursday, March 16,
2006 12:51 PM by Windows Mobile Team Blog

I've received a few questions from users who have said
that their WM5 device seems to slow down and speed...

#
Maximizing flash throughput in your apps

Thursday, March 16,
2006 1:29 PM by Windows Mobile Team Blog

Ariane in our file system team wrote a great blog entry
that describes what an application developer...

#
re: How to Programmatically Improve File System Throughput

Saturday, March 18,
2006 7:11 AM by Andreas Selle

Hi,

I've checked with many flash memory cards,
and all of them had a 512 byte block size. Does anybody know of a flash memory
card that uses a different block size?

Also, the cluster size of the FAT
system usually is bigger than 512 bytes. Wouldn't the performance be better if
all file access is aligned to the FAT cluster size instead of the flash memory
block size?

#
re: How to Programmatically Improve File System Throughput

Tuesday, March 21,
2006 5:15 PM by ce_base

Hi Andreas,
Sorry for the delayed response.  I,
personally, do not know of any flash cards that have a block size larger than
512bytes, but I did not want to assume a standard size.

You are
right that the performance would be better if the file access is aligned to the
FAT cluster size.  The FAT cluster size is a power of 2, which means that
the cluster size is a multiple of 512 bytes.  It was mentioned in the blog
that a multiple of the block size is better for performance.  Writing in
multiples of the FAT cluster size is not a bad idea either.

There
are lots of numbers that can be used as baselines for buffering: page size in
memory, cluster size in FAT, block size in flash.  This blog will (I hope)
help people understand where their performance bottlenecks are where the file
system is concerned, and help them design new applications with file system
performance in the back of their minds.

The reason the FAT cluster size
was not mentioned is that I did not want to pigeonhole the blog into being about
file access in FAT on flash memory, as what was mentioned (outside of the
directory tip) will apply to other file systems as well.

Ariane

#
The Mobile Minute 136

Friday, March 24, 2006
2:34 PM by Nino.Mobile

TGIF!
Software / Hardware

Sling Media has
released SlingPlayer for Windows Mobile (via msmobiles.com)&amp;nbsp;...

#
re: How to Programmatically Improve File System Throughput

Monday, April 17, 2006
6:28 PM by Tweakradje

After examining the fatfsd.dll from wm2003se I came to
the following registry entries that provided my device with a major performance
boost:

[HKLM\System\StorageManager\FATFS]

"BufferSize"=dword:00000040
"CacheSize"=dword:00002000 (at the cost of 2
Mb RAM)
"DLL"="fatfsd.dll" (use strings.exe from sysinternals to see these
params)
"EnableCache"=dword:00000001
"EnableCache4Way"=dword:00000001

"EnableCacheWarm"=dword:00000001
"Flags"=dword:00001006 (writethrough
bla bla)
"MaxCachedFileSize"=dword:00020000 (don't cache >128Kb files)

"Paging"=dword:00000001 (ALWAYS 1!!)
"PathCacheEntries"=dword:00000064

"UpdateAcces"=dword:00000000 (I don't need access time stamps)

[HKLM\System\StorageManager\Profiles\FlashDrv\FATFS] (could differ)

"CacheSize"=dword:00000200 (at the cost of 256 Kb RAM)

"EnableCache"=dword:00000001

Yes, it is at the cost of some RAM. Why
is/was this not implemented by default in wm2003se? How does WM2005 deals with
caching of flash memory data devices?

Cheers

#
The Curious Case Of The Four-byte Writes

Wednesday, February
14, 2007 3:34 AM by Windows Mobile Team Blog

Everybody who has worked at Microsoft for long enough has their war stories.
I'll share one of my first,

#
The Curious Case Of The Four-byte Writes

Wednesday, March 21,
2007 2:00 AM by Pocket.Net - Mobile 2.0

Everybody who has worked at Microsoft for long enough has their war stories.
I'll share one of my first,

#
The Curious Case Of The Four-byte Writes

Sunday, April 22, 2007
7:31 PM by Windows Mobile Team Blog

Everybody who has worked at Microsoft for long enough has their war stories.
I'll share one of my first,

New Comments to this post
are disabled

© 2008 Microsoft Corporation. All rights reserved. Terms of Use
|  Trademarks
|  Privacy
Statement