Click here to Skip to main content
15,922,584 members
Please Sign up or sign in to vote.
0.00/5 (No votes)
See more:
Hello,

I need to loop through a large number of files (usually around 150.000 files of approx. 48 MB each) very fast and sum up file sizes as well as file count whilst showing a progress bar for the user.

From a MS sample I learned that Parallel.ForEach seem to be the fastest way (pls have a look at 'what have you tried').

The byteCount calculates fine, while the fileCount results are false. You will surely recognize that the 'fileCount' variable isn't interlocked. Unfortunately I have no idea how to get that done properly, I run into syntax errors with whatever I try.

Could someone of you help me with the proper syntax, please?
Also I'm not sure if that's really the fastest way or if there's probably something much simpler that I could do.

Thank you,
Mick

What I have tried:

VB.NET
Parallel.ForEach(files, Function() 0, Function(f As FileInfo, loopState As ParallelLoopState, local_bytes As Long, local_files As Long)
    If (f.Attributes And FileAttributes.Directory) = 0 Then
        local_bytes += f.Length
    End If
    Return local_bytes
End Function, Sub(c)
    Interlocked.Add(total_bytes, c)   ' byteCount
    total_files += 1                  ' fileCount
    BeginInvoke(CType(Sub()
        pbVal = progBar.Value + increm
        If pbVal > 100 Then pbVal = 100
        progBar.Value = pbVal
   End Sub, Action))
End Sub)
Posted
Updated 3-Sep-20 3:22am
v3
Comments
[no name] 3-Sep-20 9:14am    
The whole concept of passing interlocked "value" variables as function parameters makes no sense. You create a common class / method that every one uses to access the "interlock", or use a lock, the variable in question; you don't pass it around.

1 solution

Try using a Tuple for the local state:
VB.NET
Parallel.ForEach(files, Function() (Length := 0L, Count := 0L), 
    Function(f As FileInfo, loopState As ParallelLoopState, index As Long, accumulator As (Length As Long, Count As Long))
        If (f.Attributes And FileAttributes.Directory) = 0 Then
            accumulator = (accumulator.Length + f.Length, accumulator.Count + 1)
        End If
        Return accumulator
    End Function,
    Sub(total As (Length As Long, Count As Long))
        Interlocked.Add(total_bytes, total.Length)
        Interlocked.Add(total_files, total.Count)
        BeginInvoke(CType(Sub()
            pbVal = progBar.Value + increm
            If pbVal > 100 Then pbVal = 100
            progBar.Value = pbVal
        End Sub, Action))
    End Sub)
Tuples - Visual Basic | Microsoft Docs[^]
 
Share this answer
 
Comments
Sonhospa 3-Sep-20 9:53am    
That's perfect - thank you, Richard!
Sonhospa 3-Sep-20 10:39am    
Hi again, sorry but I have a follow-up question... using the code now shows that the progressbar isn't working. First the result shows (for the test I display it in a message box) and AFTER this the progress bar increments. Is it at all possible to show the progress in a parallel.foreach loop?
Richard Deeming 3-Sep-20 10:41am    
It sounds like you're blocking the UI thread waiting for the Parallel.ForEach call to complete. You probably need to use a BackgroundWorker[^] to perform the work on a background thread, so that your UI has a chance to update.
Sonhospa 3-Sep-20 11:04am    
That's possible... I'll check and hope to resolve that. Thank you again!
Sonhospa 3-Sep-20 14:42pm    
Sorry that I have to ask you again... I still didn't manage to make it work. The method is called in a click-Handler like this, which should put it on a background thread:
Dim t As Task = Task.Run(Sub()
msg = MethodName(Path, "*.*", System.IO.SearchOption.AllDirectories)
End Sub)
t.Wait()

I've set a breakpoint at the "progbar.value = " line and observed that it's not reached for every file but in somewhat arbitrary intervals, which makes the calculation of the value guesswork. I'm sooooo stuck :-( that I'd really be glad about another good idea from you...

This content, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)



CodeProject, 20 Bay Street, 11th Floor Toronto, Ontario, Canada M5J 2N8 +1 (416) 849-8900