Disk I/O optimization

Disk I/O refers to the number of read and write operations performed by your
application on a physical disk or multiple disks installed in your server. Common
activities that can cause disk I/O-related bottlenecks include long-running file I/O
operations, data encryption and decryption, reading unnecessary data from database
tables, and a shortage of physical memory that leads to excessive paging activity.
Slow hard disks are another factor to consider.

To resolve disk-related bottlenecks:
- Start by removing any redundant disk I/O operations in your application.
- Identify whether your system has a shortage of physical memory, and,
if so, add more memory to avoid excessive paging.
- Identify whether you need to separate your data onto multiple disks.
- Consider upgrading to faster disks if you still have disk I/O
bottlenecks after doing all of above.

Configuration Overview
Microsoft Windows retrieves programs and data from disk. The disk
subsystem can be the most important aspect of I/O performance, but
problems can be masked by other factors, such as lack of memory.
Performance console disk counters are available within both the
LogicalDisk or PhysicalDisk objects.

Metrics
PhysicalDisk
- Avg. Disk Queue Length
- Avg. Disk Read Queue Length
- Avg. Disk Write Queue Length
- Avg. Disk sec/Read
- Avg. Disk sec/Transfer
- Disk Writes/sec


Tuning Options
If you determine that disk I/O is a bottleneck, you have a number of options:
- Defragment your disks. Use the Disk Defragmenter system tool.
- Use Diskpar.exe on Windows 2000 to reduce performance loss due to misaligned
disk tracks and sectors. You can use get the Diskpar.exe from the Windows 2000
Resource Kit.
- Use stripe sets to process I/O requests concurrently over multiple disks. The
type you use depends on your data-integrity requirements. If your applications
are read-intensive and require fault tolerance, consider a RAID 5 volume. Use
mirrored volumes for fault tolerance and good I/O performance overall. If you do
not require fault tolerance, implement stripe sets for fast reading and writing and
improved storage capacity. When stripe sets are used, disk utilization per disk
should fall due to distribution of work across the volumes, and overall throughput
should increase.
- If you find that there is no increased throughput when scaling to additional disks
in a stripe set, your system might be experiencing a bottleneck due to contention
between disks for the disk adapter. You might need to add an adapter to better
distribute the load.
- Place multiple drives on separate I/O buses, particularly if a disk has an
I/O-intensive workload.
- Distribute workload among multiple drives. Windows Clustering and
Distributed File System provide solutions for load balancing on different drives.
- Limit your use of file compression or encryption. File compression and
encryption are I/O-intensive operations. You should only use them where
absolutely necessary.
- Disable creation of short names. If you are not supporting MS-DOS for Windows
3.x clients, disable short names to improve performance. To disable short names,
change the default value of the \NtfsDisable8dot3NameCreation registry entry
(in HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Filesystem) to 1.
- Disable last access update. By default, NTFS updates the date and time stamp
of the last access on directories whenever it traverses the directory. For a large
NTFS volume, this update process can slow performance. To disable automatic
updating, create a new REG_DWORD registry entry named
NtfsDisableLastAccessUpdate in HKEY_LOCAL_MACHINE\SYSTEM\CurrentContolSet\Control\Filesystem and set its value to 1.
Reserve appropriate space for the master file table. Add the
NtfsMftZoneReservation entry to the registry as a REG_DWORD in
HKEY_LOCAL_MACHINE \SYSTEM \CurrentControlSet\Control \FileSystem.
When you add this entry to the registry, the system reserves space on the volume
for the master file table. Reserving space in this manner allows the master file
table to grow optimally. If your NTFS volumes generally contain relatively few
files that are large, set the value of this registry entry to 1 (the default).
- Typically you can use a value of 2 or 3 for moderate numbers of files, and use a value of 4 (the maximum) if your volumes tend to contain a relatively large number of files. However, make sure to test any settings greater than 2, because these greater values cause the system to reserve a much larger portion of the disk for the master file table.
- Use the most efficient disk systems available, including controller, I/O, cabling,
and disk. Use intelligent drivers that support interrupt moderation or interrupt
avoidance to alleviate the interrupt activity for the processor due to disk I/O.
- Check whether you are using the appropriate RAID configuration. Use RAID 10
(striping and mirroring) for best performance and fault tolerance. The tradeoff is
that using RAID 10 is expensive. Avoid using RAID 5 (parity) when you have
extensive write operations.
- Consider using database partitions. If you have a database bottleneck, consider
using database partitions and mapping disks to specific tables and transaction
logs. The primary purpose of partitions is to overcome disk bottlenecks for large
tables. If you have a table with large number of rows and you determine that it is
the source of a bottleneck, consider using partitions. For SQL Server, you can use
file groups to improve I/O performance. You can associate tables with file groups,
and then associate the file groups with a specific hard disk.
- Consider splitting files across hard disks. If you are dealing with extensive
file-related operations, consider splitting the files across a number of hard disks
to spread the I/O load across multiple disks.

Memory

Memory consists of physical and virtual memory. You need to consider how much memory is allocated to your application. When you evaluate memory-related bottlenecks, consider unnecessary allocations, inefficient clean up, and inappropriate caching and state management mechanisms. To resolve memory-related bottlenecks, optimize your code to eliminate these issues and then tune the amount of memory allocated to your application. If you determine during tuning that memory contention and excessive paging are occurring, you may need to add more physical memory to the server.

Low memory leads to increased paging where pages of your application’s virtual
address space are written to and from disk. If paging becomes excessive, page
thrashing occurs and intensive disk I/O decreases overall system performance.

Configuration Overview
Memory tuning consists of the following:
- Determine whether your application has a memory bottleneck. If
it has, then add more memory.
- Tune the amount of memory allocated if you can control the
allocation. For example, you can tune this for ASP.NET and
SQL Server.
- Tune the page file size.

Metrics
The performance counters help you identify memory bottlenecks. You should log these counter values to log files over a 24 hour period before you form any conclusions.

Memory
- Available MBytes
- Page Reads/sec
- Pages/sec
- Cache Bytes
- Cache Faults/sec
Server
- Pool Nonpaged Failures
- Pool Nonpaged Peak
Cache
- MDL Read Hits %

Bottlenecks
A low value of Available MBytes indicates that your system is low on physical
memory, caused either by system memory limitations or an application that is not
releasing memory. Monitor each process object’s working set counter. If Available
MBytes remains high even when the process is not active, it might indicate that the
object is not releasing memory. Use the CLR Profiler tool at this point to identify the source of any memory allocation problems. For more information, see “How To: Use
CLR Profiler” in the “How To” section of this guide.

A high value of Pages/sec indicates that your application does not have sufficient
memory. The average of Pages Input/sec divided by average of Page Reads/sec gives
the number of pages per disk read. This value should not generally exceed five pages
per second. A value greater than five pages per second indicates that the system is
spending too much time paging and requires more memory (assuming that the
application has been optimized).

Tuning Options
If you determine that your application has memory issues, your options include
adding more memory, stopping services that you do not require, and removing
unnecessary protocols and drivers. Tuning considerations include:
- Deciding when to add memory
- Page file optimization

Deciding When to Add Memory
To determine the impact of excessive paging on disk activity, multiply the values of
the Physical Disk\ Avg. Disk sec/Transfer and Memory\ Pages/sec counters. If the
product of these counters exceeds 0.1, paging is taking more than 10 percent of disk
access time. If this occurs over a long period, you probably need more memory. After
upgrading your system’s memory, measure and monitor again.

To save memory:
- Turn off services you do not use. Stopping services
that you do not use regularly saves memory and improves
system performance.
- Remove unnecessary protocols and drivers. Even idle
protocols use space in the paged and nonpaged memory
pools. Drivers also consume memory, so you should remove
unnecessary ones.

Page File Optimization
You should optimize the page file to improve the virtual memory performance of
your server. The combination of physical memory and the page file is called the
virtual memory of the system. When the system does not have enough physical
memory to execute a process, it uses the page file on disk as an extended memory
source. This approach slows performance.

To ensure an optimized page file:
- Increase the page file size on the system to 1.5 times
the size of physical memory available, but only to
a maximum of 4,095 MB. The page file needs to be at least
the size of the physical memory to allow the memory to be
written to the page file in the event of a system crash.
- Make sure that the page file is not fragmented on a given partition.
- Separate the data files and the page file to different
disks only if the disk is a bottleneck because of a lot
of I/O operation. These files should preferably be on the
same physical drive and the same logical partition. This
keeps the data files and the page file physically close to
each other and avoids the time spent seeking between two
different logical drives.

To configure the page file size
1. Open Control Panel.
2. Double-click the System icon.
3. Select the Advanced tab.
4. Click Performance Options.
5. Click Change. The Virtual Memory dialog box appears
6. Enter new values for Initial size and Maximum size. Click Set, and then click OK.

zip, zipper and 7zip log files

For various windows servers I had to decrease the amount of log files on the server itself. Therefore I decided the go with the following strategy:

• leave 3 days of log files as-is on the server


• files which are older then 3 day's need to be zipped and I keep a copy on the server


• files which are older then 60 day's can be deleted because they have already copied onto an additional storage (but this copy is a different szenario and I explain about it in a different blog post)
  

I accomplished this task with the following components:

1. 7-zip
2. forfiles.exe (you can download a rar file from my shared cache website)
3. Windows Task Scheduler
   
4. one bat file
   

The following 3 lines you multiply as much as needed for each WsSVC[n] folder and save it into a *.bat file.

@echo START - %TIME%
forfiles -p "E:\LogFiles\W3SVC1" -s -m *.log -d -3 -c "Cmd /C 7z a @FILE.zip @FILE"
forfiles -p "E:\LogFiles\W3SVC1" -s -m *.log -d -3 -c "cmd /c del @file"
forfiles -p "E:\LogFiles\W3SVC1" -s -m *.zip -d -60 -c "cmd /c del @file"
@echo DONE - %TIME%

I had to compress 24 folder with log files

All files together had a size 684 MB - well quite a lot for log files but they get very fast very big therefore I really suggest to implement this approach.

For my scripting convenience I've added the installation paths into environment path variable:

1. Open System properties
2. Click on Environment Variables
3. Search for PATH and then click edit
4. Add both full paths to 7zip and forfiles.exe
   

Now open your Task Scheduler and create a new Task:

Once you done with the configuration you should make a test run with your task.

The result is that I could save 636 MB on my hard disc on the live server. This space can be used for many other things

New feature called branchcache

in my test installation of Windows Server 2008 R2 I found today below the "Add Features Wizard" the item: "Branchcache" and I was asking myself if this is based on Velocity CTP 3 or is this an indiviual development??



BranchCache

BranchCache, available on both Windows Server 2008 R2 and Windows 7, enables client computers in a branch office to retrieve content securely and locally, instead of retrieving it from a central office server. Because branch offices are typically connected over slower WAN links, BranchCache reduces WAN traffic, and increases application responsiveness on the client computer.






BranchCache installs the services needed to provide BranchCache client functionality. It also provides BranchCache server support for the HTTP protocol, Hostet Cache, and services other then the BranchCache for network files role service.

Hyper-V Network and Windows Server 2003

If you setup a virtual machine with Windows Server 2003 Standard, Windows Server 2003 Enterprise or Windows Server 2003 Datacenter and you are missing your network then this happens because you have to install Windows Server 2003 Service Pack 2.

Without the Service Pack 2 of Windows Server 2003 the instance within Hyper-V will NOT recognize your network.

First you have to install Service Pack 2 and second you will have to install Hyper-V integration components. Due this your virtual machines guest have access to Microsoft VMBus Network Adapter.

Links:

• Windows Server 2003 Service Pack 2 (32-bit x86)
• Windows Server 2003 Service Pack 2 for x64 Editions

Note: prepare yourself with a CD you can burn since you cannot map any USB stick to Hyper-V.

After we installed Windows Server 2003 Service Pack 2 we can install: Hyper-V integration components


After the restart we simply can configure our network and everything works:



enjoy!

Windows Command Line FIND instead of GREP

People have maybe heared of grep on linux boxes... today I wanted to do a small count for the appeareance on a windows box where we host a log file with 500k rows.

P:\>find /?Searches for a text string in a file or files.

FIND [/V] [/C] [/N] [/I] [/OFF[LINE]] "string" [[drive:][path]filename[ ...]]

• /V Displays all lines NOT containing the specified string.
• /C Displays only the count of lines containing the string.
• /N Displays line numbers with the displayed lines.
• /I Ignores the case of characters when searching for the string.
• /OFF[LINE] Do not skip files with offline attribute set.
• "string" Specifies the text string to find.
• [drive:][path]filename Specifies a file or files to search.
  

If a path is not specified, FIND searches the text typed at the promptor piped from another command.

update: otherwise you can also use qgrep: http://malektips.com/xp_dos_0011.html

Windows Shortcut's - i'm loving it!

Windows Shortcut's are available from "Start->Run" and then you enter your requested shortcut.

Accessibility Controls: access.cpl
Add Hardware Wizard: hdwwiz.cpl
Add/Remove Programs: appwiz.cpl
Administrative Tools: control.exe admintools
Automatic Updates: wuaucpl.cpl
Bluetooth Transfer Wizard: fsquirt
Calculator: calc
Certificate Manager: certmgr.msc
Character Map: charmap
Check Disk Utility: chkdsk
Clipboard Viewer: clipbrd
Command Prompt: cmd
Component Services: dcomcnfg
Computer Management: compmgmt.msc
Date and Time Properties: timedate.cpl
DDE Shares: ddeshare
Device Manager: devmgmt.msc
Direct X Control Panel (if installed): directx.cpl
Direct X Troubleshooter: dxdiag
Disk Cleanup Utility: cleanmgr
Disk Defragment: dfrg.msc
Disk Management: diskmgmt.msc
Disk Partition Manager: diskpart
Display Properties: control.exe desktop
Display Properties: desk.cpl
Display Properties (w/Appearance Tab Preselected): control.exe color
Dr. Watson System Troubleshooting Utility: drwtsn32
Driver Verifier Utility: verifier
Event Viewer: eventvwr.msc
File Signature Verification Tool: sigverif
Findfast: findfast.cpl
Folders Properties: control.exe folders
Fonts: control.exe fonts
Fonts Folder: fonts
Free Cell Card Game: freecell
Game Controllers: joy.cpl
Group Policy Editor (XP Prof): gpedit.msc
Hearts Card Game: mshearts
Iexpress Wizard: iexpress
Indexing Service: ciadv.msc
Internet Properties: inetcpl.cpl
Java Control Panel (if installed): jpicpl32.cpl
Java Control Panel (if installed): javaws
Keyboard Properties: control.exe keyboard
Local Security Settings: secpol.msc
Local Users and Groups: lusrmgr.msc
Logs You Out Of Windows: logoff
Mcft Chat: winchat
Minesweeper Game: winmine
Mouse Properties: control.exe mouse
Mouse Properties: main.cpl
Network Connections: control.exe netconnections
Network Connections: ncpa.cpl
Network Setup Wizard: netsetup.cpl
Nview Desktop Manager (if installed): nvtuicpl.cpl
Object Packager: packager
ODBC Data Source Administrator: odbccp32.cpl
On Screen Keyboard: osk
Opens AC3 Filter (if installed): ac3filter.cpl
Password Properties: password.cpl
Performance Monitor: perfmon.msc
Performance Monitor: perfmon
Phone and Modem Options: telephon.cpl
Power Configuration: powercfg.cpl
Printers and Faxes: control.exe printers
Printers Folder: printers
Private Character Editor: eudcedit
Quicktime (If Installed): QuickTime.cpl
Regional Settings: intl.cpl
Registry Editor: regedit
Registry Editor: regedit32
Removable Storage: ntmsmgr.msc
Removable Storage Operator Requests: ntmsoprq.msc
Resultant Set of Policy: rsop.msc
Resultant Set of Policy (XP Prof): rsop.msc
Scanners and Cameras: sticpl.cpl
Scheduled Tasks: control.exe schedtasks
Security Center: wscui.cpl
Services: services.msc
Shared Folders: fsmgmt.msc
Shuts Down Windows: shutdown
Sounds and Audio: mmsys.cpl
Spider Solitare Card Game: spider
SQL Client Configuration: cliconfg
System Configuration Editor: sysedit
System Configuration Utility: msconfig
System File Checker Utility: sfc
System Properties: sysdm.cpl
Task Manager: taskmgr
Telnet Client: telnet
User Account Management: nusrmgr.cpl
Utility Manager: utilman
Windows Firewall: firewall.cpl
Windows Magnifier: magnify
Windows Management Infrastructure: wmimgmt.msc
Windows System Security Tool: syskey
Windows Update Launches: wupdmgr
Windows XP Tour Wizard: tourstart
Wordpad: write

shortcuts have been copied from the following location: http://khanusah.blogspot.com/2008/07/run-mmc-command.html

DUMPBIN - thats cool

get know of your assemblies with dumpbin

The Microsoft COFF Binary File Dumper (DUMPBIN.EXE) displays information about 32-bit Common Object File Format (COFF) binary files. You can use DUMPBIN to examine COFF object files, standard libraries of COFF objects, executable files, and dynamic-link libraries (DLLs).

http://support.microsoft.com/kb/177429

An interface has been developed for this nice tool: http://www.cheztabor.com/dumpbinGUI/index.htm

net start yourservice and you receive: Error 1083?

it can have many issues, but one of them - like mine, was that i had the base class: ServiceBase in a different Assembly (dll). Once I moved them to the Windows Service Project everything runs well :-)

My Error was like:

Error 1083: The executable program that this service is configured to run in does not
implement the service.


with the small change everything was fixed.

Add/Remove Programs list are missing Change and Remove buttons

in resolution which i found at the following link, not all my programms appeard:
http://www.winxptutor.com/arpbuttons.htm

so i had to download : http://www.ccleaner.com/ and then i could finally remove all my programs.


update: founded also a nice tool which shows exactly what kind of software you have installed: http://www.codeplex.com/sysadmin (you can do a lot, in case of very big AD's it's maybe not the right choice 5500++ Clients..)

How To: update within app.config file a key

today I had the pleasure to write a Setup for my project indeXus.Net Shared Cache (available on Codeplex). While others will install this WinService on various Hosts I needed a way how I can take the administration effort from the user.

In my case I had to update an appSettings Key within various *.config files and backup them into an Application Data Folder.

The solution seems to be very easy after I found all tricks....

15 static string path = Path.GetDirectoryName(Assembly.GetExecutingAssembly().CodeBase).Replace(@"file:\", "");
16 static string appPath = Environment.GetFolderPath(Environment.SpecialFolder.ApplicationData) + @"\MergeSystem\backup";

(thanks here to atif) some more coding between....

70 ///

71 /// Evaluates the and update machine IP within all Config files
72 ///

73 private static void EvaluateAndUpdateMachineIP()
74 {
75 IPHostEntry localMachineInfo = Dns.GetHostEntry(Dns.GetHostName());
76 DirectoryInfo dir = new DirectoryInfo(path);
77 foreach (FileInfo fi in dir.GetFiles("*.config"))
78 {
79 ExeConfigurationFileMap map = new ExeConfigurationFileMap();
80 map.ExeConfigFilename = fi.Name;
81
82 //Configuration config = ConfigurationManager.OpenExeConfiguration(fi.FullName);
83 Configuration config = ConfigurationManager.OpenMappedExeConfiguration(map, ConfigurationUserLevel.None);
84 if(config.AppSettings != null && config.AppSettings.Settings != null)
85 {
86 if(config.AppSettings.Settings[WinServiceCommon.Constants.ConfigServiceCacheIpAddress] != null)
87 {
88 config.AppSettings.Settings[WinServiceCommon.Constants.ConfigServiceCacheIpAddress].Value = localMachineInfo.AddressList[0].ToString();
89 config.Save(ConfigurationSaveMode.Modified);
90 ConfigurationManager.RefreshSection("appSettings");
91 }
92 }
93 }
94 }

net use and error messages

the msdn bible contains everything:

System Error Codes

The following topics provide lists of system error codes. These values are defined in the WinError.h header file.

• System Error Codes (0-499)


• System Error Codes (500-999)


• System Error Codes (1000-1299)


• System Error Codes (1300-1699)


• System Error Codes (1700-3999)


• System Error Codes (4000-5999)


• System Error Codes (6000-8199)


• System Error Codes (8200-8999)


• System Error Codes (9000-11999)


• System Error Codes (12000-15999)

enjoy it.