Click here to Skip to main content
15,846,826 members
Home / Discussions / Design and Architecture
   

Design and Architecture

 
AnswerRe: Where To Store Configuration File for This App Pin
Richard Andrew x648-Sep-23 11:33
professionalRichard Andrew x648-Sep-23 11:33 
GeneralRe: Where To Store Configuration File for This App Pin
trønderen8-Sep-23 13:19
trønderen8-Sep-23 13:19 
QuestionNeed some advice on Post-Processing large amounts of data. Pin
me@dagsunde.com3-Sep-23 22:31
professionalme@dagsunde.com3-Sep-23 22:31 
AnswerRe: Need some advice on Post-Processing large amounts of data. Pin
Pete O'Hanlon3-Sep-23 22:46
subeditorPete O'Hanlon3-Sep-23 22:46 
GeneralRe: Need some advice on Post-Processing large amounts of data. Pin
Dag Sunde3-Sep-23 23:20
Dag Sunde3-Sep-23 23:20 
GeneralRe: Need some advice on Post-Processing large amounts of data. Pin
Pete O'Hanlon4-Sep-23 0:18
subeditorPete O'Hanlon4-Sep-23 0:18 
GeneralRe: Need some advice on Post-Processing large amounts of data. Pin
Dag Sunde4-Sep-23 1:11
Dag Sunde4-Sep-23 1:11 
GeneralRe: Need some advice on Post-Processing large amounts of data. Pin
Pete O'Hanlon4-Sep-23 2:23
subeditorPete O'Hanlon4-Sep-23 2:23 
So, you have data coming in from 200 devices, 5 times a second. Let's think about the problem areas here:

  1. When you add data, at that volume, you're adding 1000 items a second. Not a problem in SQL Server.
  2. I'm assuming it's one aggregation per device every minute, so you're looking at a read of 60K rows every minute, with a write of 200 rows.

I would be tempted to invert the problem slightly. Rather than write to one source, write to two. So, write the incoming messages to SQL Server and, at the same time, write them to something like Redis Cache. Then, every minute, get your data from Redis and use that to calculate the values for the other table and write to that. This approach follows (somewhat), the CQRS so you're separating the reads and writes from each other.

The beauty is, this scales well.

GeneralRe: Need some advice on Post-Processing large amounts of data. Pin
me@dagsunde.com4-Sep-23 2:55
professionalme@dagsunde.com4-Sep-23 2:55 
AnswerRe: Need some advice on Post-Processing large amounts of data. Pin
Mircea Neacsu4-Sep-23 1:22
Mircea Neacsu4-Sep-23 1:22 
GeneralRe: Need some advice on Post-Processing large amounts of data. Pin
me@dagsunde.com4-Sep-23 1:39
professionalme@dagsunde.com4-Sep-23 1:39 
GeneralRe: Need some advice on Post-Processing large amounts of data. Pin
Mircea Neacsu4-Sep-23 1:58
Mircea Neacsu4-Sep-23 1:58 
GeneralRe: Need some advice on Post-Processing large amounts of data. Pin
me@dagsunde.com4-Sep-23 2:04
professionalme@dagsunde.com4-Sep-23 2:04 
AnswerRe: Need some advice on Post-Processing large amounts of data. Pin
jschell4-Sep-23 6:33
jschell4-Sep-23 6:33 
GeneralRe: Need some advice on Post-Processing large amounts of data. Pin
Dag Sunde4-Sep-23 10:06
Dag Sunde4-Sep-23 10:06 
Questionbest solution Pin
erummirza1-Sep-23 0:59
erummirza1-Sep-23 0:59 
AnswerRe: best solution Pin
Richard MacCutchan1-Sep-23 5:01
mveRichard MacCutchan1-Sep-23 5:01 
AnswerRe: best solution Pin
jschell4-Sep-23 6:38
jschell4-Sep-23 6:38 
QuestionQuestion on Versioning My App Pin
Kevin Marois24-Aug-23 17:12
professionalKevin Marois24-Aug-23 17:12 
AnswerRe: Question on Versioning My App Pin
Richard Deeming24-Aug-23 22:25
mveRichard Deeming24-Aug-23 22:25 
GeneralRe: Question on Versioning My App Pin
Kevin Marois25-Aug-23 8:40
professionalKevin Marois25-Aug-23 8:40 
GeneralRe: Question on Versioning My App Pin
Richard Deeming28-Aug-23 22:24
mveRichard Deeming28-Aug-23 22:24 
QuestionCard Reader Door Lock Pin
Kevin Marois23-Jul-23 13:50
professionalKevin Marois23-Jul-23 13:50 
AnswerRe: Card Reader Door Lock Pin
Peter_in_278023-Jul-23 15:18
professionalPeter_in_278023-Jul-23 15:18 
GeneralRe: Card Reader Door Lock Pin
Kevin Marois24-Jul-23 15:30
professionalKevin Marois24-Jul-23 15:30 

General General    News News    Suggestion Suggestion    Question Question    Bug Bug    Answer Answer    Joke Joke    Praise Praise    Rant Rant    Admin Admin   

Use Ctrl+Left/Right to switch messages, Ctrl+Up/Down to switch threads, Ctrl+Shift+Left/Right to switch pages.